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The  NATO  Science  and  Technology  Organization 


Science  &  Technology  (S&T)  in  the  NATO  context  is  defined  as  the  selective  and  rigorous  generation  and  application  of 
state-of-the-art,  validated  knowledge  for  defence  and  security  purposes.  S&T  activities  embrace  scientific  research, 
technology  development,  transition,  application  and  field-testing,  experimentation  and  a  range  of  related  scientific 
activities  that  include  systems  engineering,  operational  research  and  analysis,  synthesis,  integration  and  validation  of 
knowledge  derived  through  the  scientific  method. 

In  NATO,  S&T  is  addressed  using  different  business  models,  namely  a  collaborative  business  model  where  NATO 
provides  a  forum  where  NATO  Nations  and  partner  Nations  elect  to  use  their  national  resources  to  define,  conduct  and 
promote  cooperative  research  and  information  exchange,  and  secondly  an  in-house  delivery  business  model  where  S&T 
activities  are  conducted  in  a  NATO  dedicated  executive  body,  having  its  own  personnel,  capabilities  and  infrastructure. 

The  mission  of  the  NATO  Science  &  Technology  Organization  (STO)  is  to  help  position  the  Nations’  and  NATO’s  S&T 
investments  as  a  strategic  enabler  of  the  knowledge  and  technology  advantage  for  the  defence  and  security  posture  of 
NATO  Nations  and  partner  Nations,  by  conducting  and  promoting  S&T  activities  that  augment  and  leverage  the 
capabilities  and  programmes  of  the  Alliance,  of  the  NATO  Nations  and  the  partner  Nations,  in  support  of  NATO’s 
objectives,  and  contributing  to  NATO’s  ability  to  enable  and  influence  security  and  defence  related  capability 
development  and  threat  mitigation  in  NATO  Nations  and  partner  Nations,  in  accordance  with  NATO  policies. 


The  total  spectrum  of  this  collaborative  effort  is  addressed  by  six  Technical  Panels  who  manage  a  wide  range  of 
scientific  research  activities,  a  Group  specialising  in  modelling  and  simulation,  plus  a  Committee  dedicated  to 
supporting  the  information  management  needs  of  the  organization. 

AVT  Applied  Vehicle  Technology  Panel 

HFM  Human  Factors  and  Medicine  Panel 

1ST  Information  Systems  Technology  Panel 

NMSG  NATO  Modelling  and  Simulation  Group 

SAS  System  Analysis  and  Studies  Panel 

SCI  Systems  Concepts  and  Integration  Panel 

SET  Sensors  and  Electronics  Technology  Panel 


These  Panels  and  Group  are  the  power-house  of  the  collaborative  model  and  are  made  up  of  national  representatives  as 
well  as  recognised  world-class  scientists,  engineers  and  information  specialists.  In  addition  to  providing  critical 
technical  oversight,  they  also  provide  a  communication  link  to  military  users  and  other  NATO  bodies. 


The  scientific  and  technological  work  is  carried  out  by  Technical  Teams,  created  under  one  or  more  of  these  eight 
bodies,  for  specific  research  activities  which  have  a  defined  duration.  These  research  activities  can  take  a  variety  of 
forms,  including  Task  Groups,  Workshops,  Symposia,  Specialists’  Meetings,  Lecture  Series  and  Technical  Courses. 


The  content  of  this  publication  has  been  reproduced  directly  from  material  supplied  by  STO  or  the  authors. 


Published  January  2015 

Copyright  ©  STO/NATO  2015 
All  Rights  Reserved 

ISBN  978-92-837-0220-7 


Single  copies  of  this  publication  or  of  a  part  of  it  may  be  made  for  individual  use  only  by  those  organisations  or 
individuals  in  NATO  Nations  defined  by  the  limitation  notice  printed  on  the  front  cover.  The  approval  of  the  STO 
Information  Management  Systems  Branch  is  required  for  more  than  one  copy  to  be  made  or  an  extract  included  in 
another  publication.  Requests  to  do  so  should  be  sent  to  the  address  on  the  back  cover. 
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Generic  Methodology  for  Verification  and 
Validation  (GM-VV)  to  Support  Acceptance 
of  Models,  Simulations  and  Data 

(STO-TR-MSG-073) 

Executive  Summary 


Purpose 

The  Generic  Methodology  for  Verification  and  Validation  (GM-VV)  provides  a  generic  framework  to 
efficiently  develop  an  argument  to  justify  why  identified  models,  simulations,  underlying  data,  outcomes 
and  capabilities  are  acceptable  for  deployment  in  the  targeted  (intended)  operational  context  of  use. 
The  GM-VV  has  successfully  completed  the  SISO  standardization  process  through  a  GM-VV  Product 
Development  Group  (PDG)  in  order  to  provide  a  fully  accepted  VV&A  guidance  document.  In  fact, 
NATO  initiated  the  MSG-073  Task  Group  mainly  to  support  the  standardization  process  of  GM-VV. 
One  additional  objective  of  this  activity  was  to  include  the  balloted  SISO  set  of  documents  within  the 
NATO  M&S  Standards  Profile  (NMSSP).  One  of  the  key  activities  to  realize  this  MSG-073  objective  is  to 
gain  practical  experience  in  and  where  needed  provide  recommendations  to  refine  the  GM-VV  through 
case  studies. 

Scope  and  Limitations 

The  scope  of  this  V&V  methodology  is  nominally  for  NATO  M&S.  However,  the  generic  approach  taken 
herein  is  to  provide  comprehensive  guidance  documents  that  can  easily  be  adapted  and  tailored  to  other 
M&S  enterprises.  NATO  and  national  defence  establishments,  M&S  communities-of-practice  are  de  facto 
enterprises  in  this  spirit  when  the  investment,  development,  maintenance,  and  use  of  M&S  assets  are 
concerned,  and  when  V&V  of  simulations,  models  and  data  are  critical  to  increase  the  confidence  of  the 
decision-makers  in  the  results  provided  by  their  simulation  systems.  While  the  application  of  this  V&V 
methodology  is  intended  to  be  broad,  its  scope  is  targeted  to  M&S  systems  design,  development  and 
employment  process. 
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Methodologie  generale  de  verification  et  de 
validation  (GM-VV)  visant  a  soutenir  F  acceptation 
des  modeles,  simulations  et  donnees 
(STO-TR-MSG-073) 

Synthese 


Ob  jet 

La  methodologie  generale  de  verification  et  de  validation  (GM-VV)  foumit  un  cadre  general  an 
developpement  efficace  d’une  argumentation  justifiant  I’acceptabilite  de  modeles,  simulations,  donnees 
sous-jacentes,  resultats  et  capacites  identifies,  vis-a-vis  du  contexte  operationnel  cible  (prevu).  La  GM-VV 
a  suivi  avec  succes  le  processus  de  normalisation  de  la  SISO  :  elle  est  d’abord  passee  par  un  groupe  de 
developpement  de  produit  GM-VV,  qui  a  redige  un  document  d’ orientation  entierement  agree  sur  la 
verification,  la  validation  et  1’ accreditation  (VV&A).  En  realite,  I’OTAN  a  principalement  etabli  le  groupe 
de  travail  MSG-073  pour  prendre  en  charge  le  processus  de  normalisation  de  la  GM-VV.  Cette  activite 
avait  egalement  pour  but  d’inclure  le  jeu  de  documents  votes  de  la  SISO  au  sein  du  profil  des  normes  de 
M&S  de  rOTAN  (NMSSP).  L’une  des  activites  essentielles  pour  atteindre  cet  objectif  du  MSG-073  est 
d’engranger  de  I’experience  pratique  la  ou  elle  est  necessaire  pour  foumir  des  recommandations  en  vue 
d’affiner  la  GM-VV  par  des  etudes  de  cas. 

Portee  et  limites 

Cette  methodologie  de  verification  et  validation  (V&V)  ne  porte  que  sur  la  M&S  de  I’OTAN.  Toutefois, 
I’approche  generale  adoptee  ici  consiste  a  foumir  des  documents  d’orientation  complets  pouvant  etre 
facilement  adaptes  a  d’autres  entreprises  de  M&S.  L’OTAN  et  les  etablissements  de  defense  nationaux, 
reseau  d’echange  de  pratiques,  sont  de  fait  des  entreprises  en  ce  sens  lorsqu’il  s’agit  de  I’investissement, 
du  developpement,  de  I’entretien  et  de  Lutilisation  des  moyens  de  M&S  et  lorsque  la  V&V  de  simulations, 
modeles  et  donnees  est  essentielle  pour  renforcer  la  confiance  qu’accordent  les  decideurs  aux  resultats 
fournis  par  leurs  systemes  de  simulation.  Bien  que  I’application  prevue  de  la  presente  methodologie  de 
V&V  soit  large,  elle  ne  vise  que  la  conception,  le  developpement  et  le  processus  d’emploi  de  la  M&S. 
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Chapter  1  -  BACKGROUND  OF  THE  MSG-073  EFFORT 

Models  and  Simulations  (M&S)  are  developed  and  employed  as  enabling  technologies  to  support  system 
analysis,  design,  test  and  evaluation,  acquisition,  training  and  instruction,  and  many  more  areas.  Today, 
a  wide  variety  of  M&S  assets  are  in  use  across  an  even  wider  range  of  different  application  and  problem 
domains.  M&S  is  usually  applied  when  certain  user  needs  cannot  be  achieved  (e.g.  risks,  availability)  with 
the  actual  system  or  otherwise  are  achieved  more  efficiently  (e.g.  costs,  effectiveness)  than  with  the  actual 
system.  However,  in  essence,  all  M&S  assets  provide  some  sort  of  abstract  representation  of  systems 
(e.g.  entity,  phenomenon,  process)  that  are  based  on  different  types  of  approximation.  As  such,  M&S 
capabilities  cannot  fully  replace  the  actual  system  and,  more  importantly,  their  usage  introduces 
uncertainties.  In  combination  with  the  increasing  complexity  of  M&S  assets  being  developed  and  employed, 
risks  for  failures,  wrong  usage,  and  misinterpretation  of  results  are  increasingly  difficult  to  judge.  Therefore, 
the  benefits  of  using  M&S  always  come  at  some  cost,  i.e.  use  risks.  The  key  question  then  for  M&S 
stakeholders  (e.g.  user,  sponsor,  developer)  is  to  determine  which  M&S  asset  is  acceptable  for  a  particular 
intended  use,  and  which  is  not.  Verification  and  validation  (V&V)  are  the  processes  that  are  typically  used  to 
support  M&S  stakeholders  to  determine  and  assure  that  an  M&S  asset  is  acceptable  for  the  intended  use. 
Hence,  V&V  provides  information  to  be  used  in  the  acceptance  decision  process  of  M&S  stakeholders,  and 
associated  practices  such  as  M&S  accreditation  and  certification. 

The  choice  of  which  V&V  method  works  best  in  a  given  situation  depends  on  the  individual  needs  and 
constraints  of  an  M&S  organization,  project,  application  domain  or  technology.  Moreover,  V&V  usually 
requires  a  complex  mixture  of  various  activities,  methods,  tools,  techniques  and  application  domain 
knowledge,  which  are  often  tightly  coupled  with  the  M&S  development  process.  Therefore,  many  different 
approaches  to  V&V  exist  that  rely  on  a  wide  variety  of  different  V&V  terms,  concepts,  products,  processes, 
tools  or  techniques.  In  many  cases,  the  resulting  proliferation  restricts  or  even  works  against  the  transition  of 
V&V  results  ifom  one  M&S  organization,  project,  and  technology  or  application  domain  to  another. 
Furthermore,  history  shows  that  V&V  is  often  more  of  an  afterthought  than  a  built-in  part  of  an  M&S 
development,  employment  and  procurement  policy. 

In  2002,  a  European  consortium  was  created  to  address  the  W&A  issue  under  the  Western  European 
Armament  Group  (WEAG)  umbrella:  the  REWAl  (Reference  project  for  VV&A)  project  was  bom.  In  its 
final  composition  (REVVA2),  this  consortium  included  five  NATO  Member/Partner  Nations  (Canada, 
Denmark,  France,  Netherlands  and  Sweden)  with  the  task  to  complete  the  methodology  and  prepare  for 
standardization. 

The  work  of  REVVA  had  its  roots  in  both  the  UK/FR  project  MEVAS  which  was  contracted  under  the 
Anglo  French  Defence  Research  Group  (AFDRG  No  1 1)  cooperation  and  the  Intemational  Test  Operations 
Procedure  (ITOP)  Working  Group  of  Experts  (WGE  7.2). 

MEVAS  (Methode  d’Evaluation  de  Validation  et  d’Accreditation  des  Simulations)  is  the  result  of  a  Franco- 
British  study  devoted  to  the  Verification,  Validation  &  Accreditation  (VV&A)  of  the  synthetic  environment. 
MEVAS  was  intended  to  facilitate  the  establishment  of  an  organization  and  a  process  adapted  to  the 
development  of  a  simulation.  The  methodology  was  presented  in  2002  in  the  form  of  methodological  guides 
(reference  manual,  operation  manual)  stmctured  according  to  the  identified  relevant  populations:  developers, 
users  and  beneficiaries.  The  MEVAS  work,  among  other  things  developed  the  three  pillars  concept, 
a  concept  being  re-used  in  REVVA/GM-VV  (see  Chapter  5,  GM-VV  Implementation  Framework 
Dimensions  for  a  follow-up  of  this  concept). 

ITOP  is  a  four-Nation  initiative  (DEU-FRA-GBR-USA)  that  focuses  on  standards  for  test  operations.  Part  7 
focussing  on  simulation  and  V&V.  ITOP  WGE  7.2  intended  to  provide  a  standard  template  for  uniform 
documentation  in  support  of  the  exchange  of  V&V  information  among  Nations.  ITOP  made  extensive  use  of 
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V&V  cases  and  the  use  of  claims  supported  by  arguments  and  evidence  gathering,  an  approach  that  was  also 
used  as  a  baseline  for  REVVA  and  thus  for  GM-VV.  The  ITOP  group  produced  an  official  release  of  their 
work  as  n°  1-1 -002,  May  2004. 

SISO  (the  Simulation  Interoperability  Standards  Organization)  was  chosen  as  the  preferred  standardization 
organization  for  the  REVVA  methodology  that  was  the  main  objective  of  REVVA2.  According  to  SISO 
procedures  a  GM-VV  (Generic  Methodology  for  V&V)  Study  Group  (SG)  was  established  with  the 
objective  to  provide  a  path  into  the  creation  of  an  internationally  accepted  W&A  standard.  The  follow-on 
Product  Development  Group  (PDG)  of  SISO  was  tasked  to  make  all  the  required  steps  to  transform  the 
REVVA  outcomes  into  a  generic  SISO  V&V  guidance  set  of  documents. 

The  purpose  of  the  Generic  Methodology  for  Verification  and  Validation  (GM-VV)  is  to  provide  general 
applicable  guidance  for  V&V  that: 

•  Facilitates  common  understanding  and  communication  of  V&V  within  the  M&S  community; 

•  Is  applicable  to  any  phase  of  the  M&S  life-cycle  (e.g.  development,  employment,  and  reuse); 

•  Is  M&S  stakeholders’  acceptance  decision-making  process  oriented; 

•  Is  driven  by  the  M&S  stakeholders’  needs  and  M&S  use  risks  tolerances; 

•  Is  scalable  to  fit  any  M&S  scope,  budget,  resources  and  use-risks  thresholds; 

•  Is  applicable  to  a  wide  variety  of  M&S  technologies  and  application  domains; 

•  Will  result  in  traceable,  reproducible  and  transparent  evidence-based  acceptance  arguments; 

•  Can  be  instantiated  on  enterprise,  project  or  technical  levels  alike;  and 

•  Facilitates  reuse  and  interoperability  of  V&V  outcomes,  tools  and  techniques. 

GM-VV  is  not  aimed  to  replace  the  existing  V&V  approaches,  methodologies,  standards  or  policies  of  M&S 
organizations,  technology  and  application  domains;  nor  is  GM-VV’s  intent  to  substitute  common  enterprise 
or  project  management  practices  prevalent  within  M&S  client  or  supplier  organizations.  In  addition,  GM-VV 
is  not  intended  to  be  prescriptive,  in  that  it  does  not  specify  a  single  concrete  or  unique  solution  for  all  V&V 
applications.  Rather,  the  GM-VV  should  be  tailored  to  meet  the  needs  of  individual  V&V  applications. 

MSG-073  at  its  inception  in  2009  was  based  on  the  above  outline.  It  was  a  follow-up  on  earlier  efforts  within 
NATO  and  the  former  US  DoD  DMSO  (Defence  Modeling  and  Simulation  Office),  which  was  renamed  to 
US  DoD  M&SCO  (Modeling  and  Simulation  Coordination  Office).  Other  US  DoD  V&V  work  was  focussed 
on  the  development  of  IEEE  1278.4  (Recommended  Practice  for  Distributed  Interactive  Simulation  - 
VV&A)  and  IEEE  1516.4  (Recommended  Practice  for  VV&A  of  a  Federation  -  An  overlay  to  the  High 
Eevel  Architecture  Federation  Development  and  Execution  Process).  IEEE  1516.4  was  developed  with 
support  from  NATO  RTO  Task  Groups  MSG-019  and  MSG-054.  Both  the  efforts  IEEE  1278.4  and  IEEE 
1516.4  however  are  mainly  focussed  on  the  V&V  of  distributed  simulations.  On  a  national  basis,  the  US 
DoD  has  also  produced  national  guides/references  that  are  well  known  from  the  international  community 
[32]  and  have  influenced  the  work  of  GM-VV  [32], [33]. 

It  is  worth  noting  here  that  Nations  other  than  the  contributing  Nations  have  followed  the  REVVA  efforts. 
Nations  which  have  already  implemented  an  M&S  policy  in  with  direct  references  to  REVVA  (and  GM-VV) 
are  Germany  and  Australia.  The  countries  currently  participating  in  MSG-073  are  also  working  on  the 
implementation  of  GM-VV  in  their  own  organizations. 
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Chapter  2  -  OBJECTIVE  OF  THE  MSG-073  EFFORT 

In  2009,  the  NMSG  originated  a  Task  Group  (MSG-073)  to  support  the  development  of  a  set  of  guidance 
documents  on  a  generic  methodology  to  verify  and  validate  models,  simulations  and  data  which  can  be  used 
in  the  future  by  NATO  to  support  its  M&S  requirements.  The  major  objective  of  the  MSG-073  TG  was  to 
create  a  follow-on  process  for  finalizing  the  standardization  work  on  the  GM-VV  methodology.  According 
to  the  Technical  Activity  Proposal  (TAP)  and  Terms  of  Reference  (TOR)  charter,  enclosed  as  Annex  A, 
the  MSG-073  TG  was  responsible  and  engaged  in  specific  activities  with  the  following  objectives: 

•  To  finalize  the  work  on  the  VV&A  document  set  (this  task  started  under  the  European  REWA 
consortium  banner  and  is  now  under  development  by  the  SISO  GM-VV  PDG); 

•  To  guide  the  proposed  document  set  through  the  formal  SISO  standardization  process; 

•  To  participate  in  the  review,  commenting  and  balloting  efforts  needed  for  creation  of  a  new  SISO 
standard; 

•  To  assemble  confidence  in  and  fine-tune  the  methodology  on  the  basis  of  ‘real’  case  studies;  and 

•  To  provide  education  and  training  based  on  a  documentation  set  for  dissemination. 

Other  objectives  were  to  include  the  balloted  SISO  products  within  the  NATO  M&S  Standards  Profile 
(NMSSP),  AMSP-01,  and  to  produce  a  Final  Technical  Report.  The  work  of  the  Task  Group  has  built  on 
former  technical  VV&A  activities: 

•  Former  NMSG  TGs  (MSG-0 1 9  and  MSG-054); 

•  Previous  work  on  VV&A  under  the  Intemational  Test  Operations  Procedures  (ITOP)  organization; 

•  Past  work  on  the  previous  SISO  Study  Group  on  generic  Methodology  for  V&V  (GM  V&V);  and 

•  Activity  of  the  GM-VV  SISO  Product  Development  Group  (PDG). 

The  TG  has  worked  together  with  the  SISO  GM  V&V  PDG  and  was  in  direct  support  to  the  Drafting  Group 
of  the  PDG.  More  specifically  the  TG  has  been  in  charge  of  supervising  the  PDG  activities  to  guarantee  that 
NATO  and  Member/Partner  Nation’s  needs  were  addressed  in  the  PDG.  Moreover  SISO  PDGs  are  widely 
open  to  other  communities  of  interest  such  as  transport,  medicine,  etc.,  and  not  only  targeted  to  military  or 
security  purposes. 
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The  effort  deseribed  in  this  Chapter  constitutes  the  program-of-work  executed  by  the  Task  Group  MSG-073, 
which  mainly  resulted  in  the  publication  of  three  documents: 

•  GM-W  Vol.  1:  “Introduction  and  Overview”,  a  SISO  guidance  product  that  provides  an  overall 
description  of  the  methodology.  It  presents  the  core  concepts  of  the  methodology  as  well  as  how  its 
architecture  binds  them  together  to  establish  the  foundations  of  a  tailorable  implementation. 

•  GM-W  Vol.  2:  “Implementation  Guide”,  a  SISO  guidance  product  that  extends  Volume  1  by 
providing  guidance  on  how  to  apply  the  methodology.  It  unfolds  the  methodology’s  architecture  by 
elaborating  on  the  processes,  products,  interactions  among  the  roles,  and  how  to  tailor  the 
methodology. 

•  GM-VV  Vol.  3:  “Reference  Manual”,  a  SISO  reference  document  that  describes  the  foundations  of 
the  concepts,  their  dependencies  and  rationale.  This  document  is  meant  to  be  referenced  whenever  a 
deeper  technical  understanding  of  the  methodology  is  required. 

According  to  SISO,  Volume  1  and  2  are  “guidance  products”  officially  balloted  according  to  the  SISO 
Balloted  Products  Development  and  Support  Process  (reference  SlSO-ADM-003-201 1,  14  November  2011). 
Volume  3  is  a  SISO  “reference  product”  that  is  not  balloted  but  officially  approved  by  main  SISO 
committees. 


3.1  INTRODUCTION 

Both  SISO  GM-VV  PDG  and  MSG-073  efforts  are  strongly  inter-related.  MSG-073  members  were  key 
contributors  to  the  elaboration  of  SlSO-published  documents  as  either  documents  editors  or  PDG  leaders 
(see  Section  3.2.3).  For  this  reason,  many  artefacts  composing  this  TG  Final  Report  are  also  parts  of  the 
SISO  GM-VV  documents.  SISO  documents  are  freely  available  but  are  copyrighted.  For  this  reason,  all  parts 
of  SISO  documents  that  are  reproduced  within  this  final  report  with  the  agreement  of  the  SISO  Board  of 
Directors  in  compliancy  with  the  technical  cooperation  agreement  signed  between  SISO  and  the  RTO/ 
NMSG  in  2007.  The  agreement  letter  of  the  SISO  BoD  chairman  is  reproduced  in  Annex  B. 


3.2  GENERAL  DISCUSSION  OF  EFFORT  ELEMENTS 

3.2.1  Calendar  of  Meeting  Activities 

During  its  life,  the  Task  Group  met  15  times: 

•  April  2009,  in  the  NATO  R&T  Agency  (Neuilly,  FRA); 

•  September  2009,  in  University  of  Central  Florida  (Orlando,  USA); 

•  January  2010,  in  DGA  (Bourges,  FRA); 

•  March  20 1 0,  in  University  of  Central  Florida  (Orlando,  U SA); 

•  September  20 1 0,  in  University  of  Central  Florida  (Orlando,  U SA); 

•  February  20 1 1 ,  in  University  of  Bundeswehr  Munchen  (Neubiberg,  DEU); 

•  June  2011,  during  the  Euro  SIW  (The  Hague,  NED); 

•  September  201 1,  in  University  of  Central  Florida  (Orlando,  USA); 

•  February  2012,  in  FMV  (Stockholm,  SWE); 
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•  June  2012,  in  ONERA  (Toulouse,  FRA); 

•  September  2012,  in  University  of  Central  Florida  (Orlando,  USA); 

•  March  20 1 3,  in  NFR,  (Amsterdam,  NFD); 

•  June  20 1 3,  in  NATO  STO  CSO  (Neuilly,  FRA); 

•  September  2013,  in  University  of  Central  Florida  (Orlando,  USA);  and 

•  November  2013,  in  NATO  STO  CSO  (Neuilly,  FRA). 

Many  meetings  were  organized  in  the  US,  despite  the  fact  that  this  Nation  was  not  part  of  this  technical 
activity,  for  organisational  reasons:  all  these  meetings  were  co-located  with  SISO  workshops  (SIWs). 

3.2.2  Task  Group  Work-Product  Generation 

All  products  of  the  Task  Group  are  linked  with  the  TG  objectives.  In  addition  to  the  three  SISO  documents 
published  as  official  SISO  documents  but  freely  available  to  the  M&S  community,  other  products  relate  to 
the  TG  objectives  as  expressed  in  the  Terms  of  Reference  of  the  MSG-073  technical  activity: 

•  To  finalize  the  work  on  a  VV&A  document  set  under  development  by  the  SISO  GM-V&V  Product 
Development  Group; 

•  To  guide  the  proposed  document  set  through  the  formal  SISO  standardization  process; 

•  To  participate  in  the  review,  comment  and  balloting  efforts  for  these  new  SISO  standards; 

•  To  assemble  confidence  in  and  fine-tune  the  methodology  on  the  basis  of  ‘real’  case  studies;  and 

•  To  provide  education  and  training  based  on  a  documentation  set  for  dissemination. 

Objectives  1  to  3  correspond  to  the  supporting  activity  of  the  SISO  GM-VV  PDG  (see  Section  3.2.3)  that  has 
resulted  in  the  three  SISO  documents. 

Objective  4:  Chapter  7  describes  case  studies  as  known  at  the  time  of  completion  of  this  report. 

Objective  5  has  been  achieved  with  the  production  of  three  tutorials  at  different  levels: 

•  One  high-level  presentation  of  the  methodology  (half-an-hour  introduction)  dedicated  to  managers 
and  M&S  policy-level  individuals; 

•  Introductory  Tutorial  (1  hour  to  90  minutes):  this  tutorial  was  prepared  by  NFD  and  has  already 
been  introduced  in  some  non-NATO  events  such  as  SISO  Workshops,  ITEC  and  FITSEC 
conferences;  and 

•  A  2-day  tutorial  initially  developed  by  France  for  the  education  and  training  of  GM-VV  users. 
This  longer  tutorial  is  the  foundation  for  a  STO  Fecture  Series. 

3.2.3  Coordination  with  SISO  GM-W  PDG 

According  to  its  TOR,  MSG-073  had  to  support  the  SISO  Product  Development  Group  that  has  the  mission 
to  develop  and  ballot  the  GM-VV  documents.  The  MSG-073  acted  has  a  “management  group”  for  the  SISO 
GM-VV  PDG. 

MSG-073  provided  the  chair,  one  co-vice-chair  and  the  secretary  of  the  PDG.  Main  members  of  the  GM-VV 
Drafting  Group  (DG)  including  both  DG  Editors  were  members  of  the  MSG-073  TG.  Nearly  all  MSG-073 
members  are  also  members  of  the  SISO  PDG:  this  ensured  that  concepts  developed  by  the  MSG-073  TG 
were  well  taken  into  account  in  the  SISO  PDG  production,  while  MSG-073  members  were  voting  within  the 
SISO  PDG  when  decisions  had  to  be  taken. 
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3.3  CONCLUSION 

As  far  as  the  Task  Group  can  judge,  it  has  completed  its  initial  5  objectives.  In  addition,  MSG-073  dealt  with 
specifications  of  supporting  tools  that  was  not  explicitly  expressed  in  its  TOR. 

Concerning  the  SISO  activity,  the  support  to  GM-VV  standardization  is  a  permanent  future  activity  and  a 
Task  Group  has  a  limited  life-time.  The  NMSG  should  decide  what  specific  actions  they  should  initiate  to 
support  this  future  effort. 

A  Product  Support  Group  (PSG)  was  instantiated  by  SISO  in  December  2013.  In  accordance  with  SISO 
regulations,  a  PSG  serves  as  the  central  point  for  interpretation  of  product  language,  providing  help  desk 
support,  and  accepting,  developing,  and  maintaining  problem  and  change  reports  to  support  future  product 
revisions. 
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Chapter  4  -  GM-VV  CONCEPTUAL  FRAMEWORK* 

4.1  INTRODUCTION 

The  GM-VV  provides  a  teehnical  framework  that  foeuses  on  M&S  V&V  praetiees.  Though  interrelated, 
acceptanee  decision  processes  and  associated  practices  such  as  M&S  accreditation  and  certification  are 
outside  the  scope  of  the  methodology. 


GM-VV  attains  its  generic  quality  from  a  technical  framework  that  consists  of  three  sub-parts: 
the  conceptual,  implementation  and  tailoring  framework  (Figure  4-1).  This  framework  is  rooted  in 
established  international  standards  and  other  related  practices.  The  conceptual  framework  provides  the 
terminology,  concepts  and  principles  to  facilitate  communication  and  a  common  understanding  and 
execution  of  V&V  within  an  M&S  context.  The  implementation  framework  translates  these  concepts  and 
principles  into  a  set  of  generic  building  blocks  to  develop  consistent  V&V  solutions  for  an  individual  M&S 
organization,  project,  and  technology  or  application  domain.  GM-VV  provides  a  tailoring  framework  that 
utilizes  these  building  blocks  to  develop  and  cost-efficiently  apply  such  V&V  application  instances.  As  such, 
the  GM-VV  provides  a  high-level  framework  for  developing  concrete  V&V  solutions  and  conducting  V&V, 
into  which  lower-level  practices  (e.g.  tools,  techniques,  tasks,  acceptability  criteria,  documentation 
templates)  native  to  each  individual  M&S  organization,  project,  and  technology  or  application  domain  can 
easily  be  integrated. 


Figure  4-1:  GM-W  Technical  Framework  Design  and  Operational  Use  Concept. 


4.2  GM-VV  CONCEPTUAL  FRAMEWORK 

This  framework  provides  fundamental  and  general  applicable  terminology,  semantics,  concepts  and 
principles  for  V&V.  The  purpose  of  the  framework  is  to  facilitate  communication,  understanding  and 
implementation  of  V&V  across  and  between  different  M&S  contexts  (e.g.  organizations,  application 
domains,  standards,  technologies).  The  framework  is  the  foundation  upon  which  the  GM-W 
implementation  framework  rests. 


’  Parts  of  this  chapter  are  taken  from  SISO-GUIDE-001.1-2012,  “GM-W  Vol.l:  Introduction  and  Overview”  (Copyright  2012 
by  Simulation  Interoperability  Standards  Organization  -  SISO)  [1]. 
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4.2.1  Links  to  System  Engineering 

Within  the  GM-VV,  M&S  systems  are  considered  to  be  Systems-of-Systems  (SoS)  that  have  a  life-cycle  and 
are  subject  to  system  engineering  practices.  Moreover,  models  and  simulations  are  considered  to  be  part  of  a 
larger  system  in  which  they  are  used.  From  this  perspective,  M&S  is  a  systems  engineering  specialization. 
Verification  and  Validation  (V&V)  are  an  intrinsic  part  of  the  systems  engineering  process  [23], [27], [28], 
[29].  Therefore,  the  GM-VV  considers  the  V&V  of  M&S  as  a  specialization  of  systems  engineering  V&V. 
Hence,  the  GM-VV  can  be  integrated  with,  complement  or  extend  the  V&V  processes  within  such  existing 
systems  engineering  methodologies  or  standards. 

4.2.2  M&S-Based  Problem  Solving  Approach 

The  basic  premise  of  the  GM-VV  is  that  models  and  simulations  are  always  developed  and  employed  to 
fulfil  the  specific  needs  of  their  end  users  (e.g.  trainers,  analysts,  decision-makers).  Modeling  and  simulation 
is  thus  considered  to  be  a  problem  solving  process  that  transforms  a  simple  statement  of  an  end  user’s  need 
into  an  M&S-based  solution  for  the  problem  implied  in  the  need.  The  GM-VV  assumes  that  V&V  always 
takes  place  within  such  larger  context.  This  context  is  abstracted  by  means  of  defining  four  interrelated 
worlds  (Figure  4-2).  Together,  these  four  worlds  define  a  generic  life-cycle  and  process  view  of  M&S-based 
problem  solving.  A  view  that  serves  as  a  common  basis,  in  which  V&V  for  M&S  (e.g.  concepts,  principles, 
processes,  products,  techniques)  can  be  understood,  developed  or  applied. 


Real  World 

Problem  World 

M&S  World 

Product  World 
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M&S 
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M&SHW/SW 
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.  I 
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Figure  4-2:  Four  Worlds  View  of  M&S-Based  Problem  Solving. 

In  the  four-world’s  view  of  M&S-based  problem  solving.  Real  World  operational  needs  are  translated  into 
an  M&S  intended  use  statement  in  the  Problem  World.  In  the  M&S  World  this  M&S  intended  use  serves  as 
the  basis  from  which  the  M&S  requirements  are  set  for  an  M&S  System.  The  M&S  system  is  developed 
within  the  Product  World,  which  involves  setting  low-level  requirements  for  each  element  in  the  M&S 
system.  M&S  employment  in  the  M&S  World  comprises  well-controlled  operation  of  the  M&S  system. 
The  M&S  results  that  come  from  the  M&S  system  operation  (i.e.  simulation  execution)  are  used  in  the 
problem  solving  process  within  the  problem  world.  Finally,  the  M&S-based  solution  (e.g.  new  product, 
trained  person,  and  analysis  report)  is  transferred  to  the  real  world  where  it  is  exploited  in  the  operational 
environment.  When  this  M&S  problem  solving  process  is  properly  executed,  the  resulting  solution  should 
satisfy  the  originally  identified  needs  with  a  minimal  level  of  (use)  risk  in  the  Real  World. 

The  M&S  system,  M&S  requirements,  M&S  results  and  other  development  artifacts  (e.g.  conceptual  model, 
software  design,  code)  are  thus  always  directed  toward  contributing  to  and  satisfying  the  Real  World 
operational  needs.  The  degree  of  success  of  such  M&S  in  satisfying  these  needs  depends  on  how  well  they 
are  specified,  designed,  developed,  integrated,  tested,  used,  and  supported.  These  M&S  activities  require  the 
contribution  of  individuals  or  organizations  that  have  a  vested  interest  in  the  success  of  the  M&S  asset. 
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either  directly  or  indirectly.  An  individual  or  organization  with  such  interest  is  referred  to  in  GM-VV  as  a 
stakeholder.  Stakeholders  can  play  one  or  more  roles  in  each  of  the  four  worlds  such  as  M&S  user/sponsor, 
supplier,  project  manager,  software  developer,  operator,  customer  or  Subject-Matter  Expert  (SME). 
Depending  upon  their  role,  stakeholders  may  hold  different  responsibilities  in  the  M&S  life-cycle  processes, 
activities  or  tasks. 

Notice  that  the  four-world  view  is  not  intended  to  be  prescriptive  or  to  replace  alternative,  more  detailed 
M&S  life-cycle  and  process  implementations  (e.g.  FEDEP  [8],  DSEEP  [7]  that  are  required  or  used  by 
organizations.  Such  concrete  implementations  can  be  considered  as  tailored  instances  of  this  abstract  four- 
world  view. 

4,2,3  V&V  Problem  Solving  Approach 

Within  the  four-world  context,  stakeholders  exist  who  are  responsible  for  making  acceptance  decisions  on 
the  use  of  M&S.  Within  the  GM-VV,  these  stakeholders  are  referred  as  V&V  User/Sponsor.  In  this  context, 
the  V&V  User/Sponsor  could  be  an  M&S  User/Sponsor,  Accreditation  Authority  or  any  other  domain 
specific  role  that  uses  the  outcomes  of  the  V&V.  V&V  Users/Sponsors  face  the  problem  of  having  to  make  a 
judgment  on  the  development  and  suitability  of  the  M&S  system  or  results  for  an  intended  use.  The  key  issue 
here  is  that  it  is  not  possible  to  demonstrate  with  absolute  certainty  that  the  M&S  system  or  results  will  meet 
the  Real  World  needs  prior  to  its  actual  use.  Consequently,  there  is  always  a  probability  that  the  M&S-based 
solution  is  not  successful  when  used  (i.e.  fails).  Such  a  failure  would  result  in  an  undesirable  impact 
(i.e.  a  risk)  on  the  operational  environment.  Therefore,  an  M&S  system  or  result  is  only  acceptable  to  the 
V&V  User/Sponsor  if  he  or  she  has  sufficient  confidence  that  the  use  of  an  M&S  system  or  result  satisfies 
the  Real  World  needs  without  posing  unacceptable  risks  (e.g.  costs,  liabilities).  This  M&S  acceptability  is 
something  relative  to  different  V&V  Users/Sponsors:  what  is  acceptable  to  one  V&V  User/Sponsor  may  not 
be  acceptable  for  another.  The  V&V  User/Sponsor’s  decision-making  process  therefore  requires  appropriate 
evidence-based  arguments  to  justify  his  or  her  acceptance  decision. 

The  basic  premise  of  GM-VV  is  that  V&V  are  performed  to  collect,  generate,  maintain  and  reason  with 
a  body  of  evidence  in  support  of  the  V&V  Users/Sponsors  acceptance  decision-making  process. 
Here,  validation  is  referred  to  as  the  process  that  establishes  the  V&V  User/Sponsor’s  confidence  as  to 
whether  or  not  they  have  built  or  procured  the  right  M&S  system  or  result  for  the  intended  use  (i.e.  M&S 
validity).  In  other  words  “Did  we  build  the  right  M&S  system?”  To  ensure  that  the  M&S  system  or  results  at 
delivery  can  be  demonstrated  to  be  valid,  it  is  necessary  to  ensure  that  the  M&S  system  is  built  and 
employed  in  the  right  manner.  Here  verification  is  referred  to  as  the  process  of  establishing  V&V  User/ 
Sponsors  confidence  in  whether  the  evolving  M&S  system  or  result  is  built  right  (i.e.  M&S  correctness). 
In  other  words  “Did  we  build  the  M&S  system  right?”  The  GM-VV  considers  V&V  as  a  specific  problem 
domain  of  M&S  with  its  own  needs,  objectives  and  issues.  This  domain  is  referred  to  as  the  V&V  World 
(Figure  4-3). 
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Figure  4-3:  V&V  World  and  Four-World  Interfacing. 


The  V&V  world  groups  the  products,  processes  and  organizational  aspects  that  are  needed  to  develop  an 
acceptance  recommendation  that  can  be  used  by  the  V&V  User/Sponsor  in  his  or  her  acceptance  decision 
procedure(s).  This  recommendation  included  in  a  V&V  report  is  the  key  deliverable  of  a  V&V  effort  and 
contains  evidence-based  arguments  regarding  the  acceptability  of  an  M&S  system  or  results.  Here  the 
GM-VV  premise  is  that  the  acceptance  decision  itself  is  always  the  responsibility  of  the  V&V  User/Sponsor 
and  decision  procedure(s)  may  involve  trade-off  aspects  beyond  the  V&V  effort  scope. 

The  development  of  an  acceptance  recommendation  in  the  V&V  world  is  driven  by  the  V&V  needs  that  are 
traceable  to  the  V&V  User/Sponsor’s  acceptance  decision  or  procedure(s)  needs  (e.g.  budget,  responsibilities, 
risks,  liabilities).  Therefore,  the  extent,  rigor  and  timeframe  of  a  V&V  effort  depend  on  these  needs. 
Depending  on  these  needs,  the  V&V  effort  could  span  the  whole  or  specific  M&S  life -cycle  phase  of  the 
four  worlds;  could  focus  on  one  specific  or  multiple  (intermediate)  M&S  products;  and  should  match  the 
development  paradigm  that  was  used  (e.g.  waterfall,  spiral).  Each  case  may  require  a  separate  acceptance 
recommendation  with  its  own  scope  and  development  timeline.  Moreover,  the  way  the  V&V  effort  interacts 
with  the  four  M&S-based  problem  worlds  also  varies  from  case  to  case.  These  mutual  dependencies  are 
depicted  in  Figure  4-3  with  bi-directional  arrows  that  interface  the  V&V  world  with  each  of  the  four  M&S- 
based  problem  solving  worlds.  Two  classical  types  of  V&V  that  can  be  identified  based  on  the  time  frame  of 
their  execution  are  [23], [24], [25], [26]: 

•  Post-Hoc  V&V:  V&V  conducted  in  retrospect  on  an  M&S  system  after  development  or  on  M&S 
results  after  M&S  system  employment;  and 

•  Concurrent  V&V:  V&V  conducted  in  prospective  throughout  the  whole  M&S  life -cycle  to  manage 
and  improve  the  quality  of  newly  developed  M&S  systems  or  results. 

The  GM-VV  supports  both  V&V  time  frames  but  is  not  limited  to  these  distinct  types.  A  V&V  effort  can  be 
post-hoc,  concurrent,  iterative,  recursive  or  even  be  a  recurrent  effort  in  the  case  where  legacy  M&S  products 
are  updated  or  reused  for  a  different  intended-use. 
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4,2,4  Acceptance  Recommendation,  Acceptability  Criteria  and  Evidential  Quality 

The  objective  of  a  V&V  effort  is  to  develop  evidence  upon  which  an  acceptance  recommendation  is  based. 
This  V&V  objective  is  articulated  as  an  acceptance  goal.  This  high-level  goal  should  be  translated  into  a  set 
of  concrete  and  assessable  acceptability  criteria  for  the  M&S  system  or  result(s).  Relevant  and  convincing 
evidence  should  then  be  collected  or  generated  to  assess  the  satisfaction  of  these  criteria.  When  it  is 
convincingly  demonstrated  to  what  extent  the  M&S  system  or  result(s)  does  or  does  not  satisfy  all  these 
acceptability  criteria,  a  claim  can  be  made  on  whether  or  not  the  M&S  system  or  result(s)  is  acceptable  for  its 
intended  use  (i.e.  acceptance  claim). 

GM-VV  identifies  four  types  of  M&S  properties  for  which  acceptability  criteria  could  be  set  (Figure  4-4): 

•  UtOity:  This  property  refers  to  the  extent  to  which  the  M&S  system  or  result(s)  is  useful  in  solving 
the  M&S  user/sponsor’s  needs.  Utility  properties  could  comprise  sub-types  such  as  M&S  value 
(e.g.  measures  of  effectiveness,  measures  of  performance),  cost  (e.g.  money,  time)  and  use  risks 
(e.g.  impact,  ramifications); 

•  Validity:  This  property  refers  to  the  extent  to  which  the  M&S  system’s  representation  corresponds 
to  the  simulated  simuland  (i.e.  system  of  interest)  from  the  perspective  of  the  intended  use.  The  level 
of  validity  impacts  the  utility;  and 

•  Correctness:  This  property  refers  to  the  extent  to  which  the  M&S  system  implementation  conforms 
to  its  specifications  (e.g.  conceptual  model,  design  specification);  and  is  free  of  design  and 
development  defects  (e.g.  semantic  errors,  syntactic  errors,  numerical  errors,  user  errors).  The  level 
of  correctness  impacts  both  validity  and  utility. 

•  V&V  Quality:  This  property  refers  to  how  well  the  V&V  effort  is  performed  (e.g.  rigor)  with  regard 
to  developing  the  acceptability  criteria,  collecting  evidence,  and  assessing  to  what  extent  the  M&S 
satisfy  the  acceptability  criteria  (e.g.  evidential  value,  strength). 

Typical  examples  of  V&V  quality  properties  are  the  completeness,  consistency,  unambiguous  and 
relevance  of  the  acceptability  criteria  or  their  supporting  items  of  evidence.  In  the  process  of 
collecting  or  generating  evidence,  quality  properties  could  comprise  independence  of  applied  V&V 
techniques  or  persons,  knowledge  gaps  and  uncertainties  of  referent  data  for  the  simuland  [22],  skill 
level  of  V&V  personnel,  and  reliability  and  repeatability  of  V&V  techniques.  Relevance  and 
warrants  for  any  assumption  made  in  a  V&V  effort  could  also  be  addressed  in  the  form  of  quality 
properties. 

These  four  types  of  M&S  properties  include  but  not  limited  to  capability,  accuracy,  usability  and  fidelity 
[17], [18].  To  make  an  acceptance  decision,  the  V&V  User/Sponsor  needs  to  know  whether  the  M&S  system 
or  results  are  (un)acceptable,  as  well  as  the  evidential  value  of  this  acceptance  claim  (i.e.  strength).  The 
required  evidential  strength  to  establish  sufficient  trust  in  such  a  claim  depends  on  the  use  risks  and  the  V&V 
User/Sponsor  responsibilities  (i.e.  liability).  The  convincing  force  that  can  be  placed  on  such  a  claim  depends 
on  the  quality  of  the  whole  V&V  effort.  For  this  purpose,  the  GM-VV  identifies  quality  properties  that  can 
be  associated  with  identifying  and  defining  the  acceptability  criteria;  and  developing  convincing  evidence  for 
demonstrating  their  satisfaction  (Figure  4-4). 
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Figure  4-4:  Utility,  Validity,  Correctness  and  V&V  Quality  Criteria  Relationships. 


The  defined  acceptability  criteria,  the  collected  evidence  and  assessment  of  the  satisfaction  of  these  criteria 
are  the  basis  for  developing  the  arguments  underlying  the  acceptance  claim.  This  acceptance  claim  provides 
the  V&V  User/Sponsor  with  a  recommendation  regarding  the  acceptability  of  the  M&S  system  or  result  for 
the  intended  use.  In  practice,  an  acceptance  recommendation  is  not  necessarily  just  a  yes  or  no  claim, 
in  the  sense  that  an  M&S  system  or  results  can  be  accepted  only  if  it  meets  all  of  the  acceptability  criteria 
and  cannot  be  accepted  if  it  does  not.  Meeting  all  the  acceptability  criteria  means  the  claim  can  be  made  that 
the  M&S  system  or  result  should  be  accepted  to  support  the  intended  use  without  limitations.  In  case  not  all 
acceptability  criteria  are  met,  alternative  weaker  acceptance  claims  with  underlying  arguments  can  be 
constructed.  Such  alternative  acceptance  claims  could,  for  example,  provide  recommendations  regarding 
conditions  or  restrictions  under  which  the  M&S  system  or  result  can  still  be  used  (i.e.  limit  the  domain  of 
use);  or  on  modifications  that,  when  implemented,  will  lead  to  an  unconditionally  acceptable  M&S  system  or 
results  for  the  intended  use.  Another  rationale  for  alternative  acceptance  claims  is  when  convincing  or 
sufficient  evidence  is  lacking  (e.g.  access  to  data  prohibited,  or  referent  system  unavailable  for  testing). 
In  any  case,  an  acceptance  recommendation  always  requires  well-structured  supporting  arguments  and 
evidence  for  the  V&V  User/Sponsor  to  make  the  right  acceptance  decision.  Depending  on  the  identified 
M&S  use  risk,  the  V&V  User/Sponsor  can  also  decide  not  to  take  any  actions  when  not  all  acceptability 
criteria  are  met  by  the  M&S  system.  In  that  case,  the  V&V  User/Sponsor  simply  accepts  the  risks  associated 
with  the  M&S  system  use. 

4,2,5  V&V  Argumentation  Approach:  Structured  Reasoning  with  Arguments 

Developing  an  acceptance  recommendation  that  meets  the  V&V  User/Sponsor  needs  usually  involves  the 
identification  and  definition  of  many  interdependent  acceptability  criteria,  particularly  for  large-scale  and 
complex  M&S  systems  or  for  M&S-based  solutions  used  in  safety-critical,  real-world  environments. 
Demonstrating  the  satisfaction  of  acceptability  criteria  requires  evidence.  Collecting  the  appropriate  evidence 
is  not  always  simple  and  straight-forward,  or  even  not  always  possible  due  to  various  practical  constraints 
(e.g.  safety,  security,  costs,  schedule).  In  many  cases,  the  collected  evidence  comprises  a  large  set  of 
individual  items  or  pieces  of  evidence  that  may  be  provided  in  different  forms  or  formats,  and  may  originate 
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from  various  sources  (e.g.  historical,  experimental  data,  SME  opinion).  Moreover,  the  strength  of  each  item 
of  evidence  may  vary  and  the  total  set  of  collected  evidence  may  even  contain  contradicting  items 
of  evidence  (i.e.  counter  evidence).  The  quality  of  this  effort  determines  the  value  of  an  acceptance 
recommendation  for  the  V&V  User/Sponsor.  Therefore,  the  arguments  underlying  an  acceptance 
recommendation  should  be  developed  in  a  structured  manner  using  a  format  where  the  reasoning  is  traceable, 
reproducible  and  explicit.  Alternative  approaches  to  implement  such  reasoning  exist  and  may  be 
incorporated  within  the  GM-VV  technical  framework  to  tailor  it  the  specific  needs  of  an  M&S  organization 
or  domain.  An  example  of  such  an  approach  is  the  V&V  goal-claim  network  approach  (Figure  4-5)  as 
developed  by  the  WEAG  REVVA  consortium  and  the  NATO  NSMG-073  Task  Group.  A  V&V  goal-claim 
network  is  an  information  and  argumentation  structure  rooted  in  both  goal -oriented  requirements  engineering 
and  claim-argument-evidence  safety  engineering  principles. 

The  V&V  Goal  Network  The  V&V  Claim  Network 


Acceptance 
Planning  Assessment 


V&V 

lanning  Assessmen 


V&V 

Experimental  Frame 


V&V  Execution 


V&V  Results 


Figure  4-5:  V&V  Goal  -  Claim  Network. 


Figure  4-5  provides  an  abstract  illustration  of  a  V&V  goal-claim  network.  The  left  part  of  the  goal-claim 
network  is  used  to  derive  the  acceptability  criteria  from  the  acceptance  goal;  and  deriving  solutions  for 
collecting  evidence  to  demonstrate  that  the  M&S  asset  satisfies  these  criteria  as  indicated  by  the  top-down 
arrows  (Figure  4-5).  The  acceptance  goal  reflects  the  V&V  needs  and  scope  (e.g.  simuland,  intended  use). 
Evidence  solutions  include  the  specification  of  tests/experiments,  referent  for  the  simuland  (e.g.  expected 
results,  observed  real  data),  methods  for  comparing  and  evaluating  the  test/experimental  results  against  the 
referent.  Collectively,  they  specify  the  design  of  the  V&V  experimental  frame  used  to  assess  the  M&S 
system  and  its  results.  When  implemented,  the  experimental  frame  produces  the  actual  V&V  results.  After  a 
quality  assessment  (e.g.  for  errors,  reliability,  strength),  these  results  can  be  used  as  the  items  of  evidence  in 
the  right  part  of  the  goal-claim  network.  These  items  of  evidence  support  the  arguments  that  underpin  the 
acceptability  claims.  An  acceptability  claim  states  whether  a  related  acceptability  criterion  has  been  met  or 
not. 

Acceptability  claims  provide  the  arguments  for  assessing  whether  or  to  what  extent  the  M&S  system 
and  its  results  are  acceptable  for  the  intended  use.  This  assessment,  as  indicated  by  the  bottom-up  arrows 
(Figure  4-5),  results  in  an  acceptance  claim  inside  the  V&V  goal-claim  network.  As  such  a  V&V  goal-claim 
network  encapsulates  structures  and  consolidates  all  underlying  evidence  and  argumentation  necessary 
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for  developing  an  appropriate  and  defensible  acceptance  recommendation.  The  circular  arrows  in  Figure  4-5 
represent  the  iterative  nature  of  developing  a  V&V  goal-claim  network  during  planning,  execution  and 
assessment  phases  of  a  V&V  effort. 

4,2,6  V&V  Organizational  and  Management  Approach 

In  order  to  facilitate  efficient  and  high  quality  V&V,  the  V&V  effort  inside  the  V&V  world  should  be 
executed  in  a  controlled  and  organized  way.  The  basic  premise  of  the  GM-VV  is  that  the  acceptance 
recommendation  for  an  M&S  asset  is  developed  and  delivered  by  means  of  a  managed  project.  Moreover, 
GM-VV  assumes  that  V&V  is  conducted  by  a  person,  a  team  of  people  or  a  dedicated  organization  with 
assigned  responsibilities,  obligations  and  functions.  Therefore,  GM-VV  identifies  three  organizational  levels 
at  which  V&V  efforts  can  be  considered.  In  order  of  the  lowest  to  the  highest  organizational  level  these 
levels  are: 

•  Technical  Level:  Concerns  the  engineering  aspects  of  a  V&V  effort  that  are  necessary  to  develop 
and  deliver  an  acceptance  recommendation; 

•  Project  Level:  Concerns  the  managerial  aspects  related  to  the  proper  execution  of  the  technical 
actions  of  a  V&V  effort;  and 

•  Enterprise  Level:  Concerns  the  strategic  and  enabling  aspects  to  establish,  direct  and  support  the 
execution  or  business  environment  for  V&V  efforts. 

The  core  GM-VV  concept  on  the  V&V  project  level  is  the  concept  of  a  managed  project.  A  V&V  project  can 
be  viewed  as  a  unique  process  comprised  of  coordinated  and  controlled  activities  that  address:  V&V  effort 
planning  in  terms  like  cost,  timescales  and  milestones;  measuring  and  checking  progress  against  this 
planning;  and  selecting  and  taking  corrective  actions  when  needed.  A  V&V  project  could  be  a  separate 
project  alongside  the  M&S  project  of  which  the  M&S  asset  is  part,  or  be  an  integral  part  of  this  M&S  project 
itself  (e.g.  sub-project,  work  package).  A  separate  V&V  project  is  particularly  relevant  in  the  case  when  a 
level  of  independence  must  be  established  between  the  M&S  development  and  V&V  team/organization. 
On  the  V&V  project  level,  GM-VV  also  provides  derived  concepts  such  as  a  V&V  plan  and  report  to 
manage  the  technical  V&V  work. 

The  core  GM-VV  concept  on  the  V&V  enterprise  level  is  the  concept  of  an  enterprise  entity.  A  V&V 
enterprise  entity  can  be  viewed  as  an  organization  that:  establishes  the  processes  and  life-cycle  models  to  be 
used  by  V&V  projects;  initiates  or  defers  V&V  projects;  provides  resources  required  (e.g.  financial,  human, 
equipment);  retains  reusable  knowledge  and  information  from  current  V&V  projects;  and  leverages  such 
knowledge  and  information  from  previous  V&V  projects.  The  V&V  enterprise  provides  the  environment  in 
which  V&V  projects  are  conducted.  GM-VV  defines  two  types  of  enterprise  entities: 

•  V&V  Client:  The  person  or  organization  that  acquires  V&V  products  or  services;  and 

•  V&V  Supplier:  The  person  or  organization  that  develops  and  delivers  V&V  products  or  services. 

A  V&V  agreement  is  arranged  between  a  V&V  client  and  V&V  supplier  to  provide  products  and/or  services 
that  meet  the  V&V  client’s  needs.  Both  these  V&V  entities  could  be  organizations  (e.g.  companies)  separate 
from  the  organization  that  develops  or  acquires  M&S  or  it  could  be  different  units  (e.g.  department,  division, 
group)  within  a  single  M&S  supplier  or  client  organization.  Typically,  a  separate  V&V  supplier  is  an 
organization  that  has  the  provision  of  independent  V&V  products  and  services  to  external  V&V  clients  as  its 
core  business.  Though  depending  on  their  business  model,  an  M&S  supplier  or  client  organization  could 
have  their  own  V&V  supplier  entity  that  may  provide  V&V  services  and  products  to  internal  and  external 
V&V  clients  alike. 
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4,2,7  V&V  Levels  of  Independence:  Acceptance,  Certification  and  Accreditation 

An  independent  V&V  (IV&V)  authority  is  often  described  as  an  organization  or  a  person  that  is  employed  to 
conduct  V&V,  independent  of  the  developer’s  team  or  organization.  The  need  for  IV&V  is  mostly  driven  by: 

•  Risks  and  liabilities  taken  by  the  V&V  User/Sponsor’s  acceptance  decision; 

•  Level  of  trust  the  V&V  User/Sponsor  has  in  the  M&S  developer; 

•  Authoritative  policies  and  regulations  that  may  demand  independent  V&V  for  the  M&S  intended 
use;  and 

•  Lack  of  specialist  skills,  tools  and  techniques  by  user,  sponsor  or  developer  to  perform  V&V. 

In  this  context,  the  terms  “certification”  and  “accreditation”  are  often  used.  Certification  is  the  process  of 
providing  a  written  guarantee  that  a  (M&S)  system  is  acceptable  for  operational  use  [4].  Accreditation  has 
two  connotations.  Accreditation  is  the  official  certification  that  a  (M&S)  system  is  acceptable  for  use  for  a 
specific  purpose,  as  used  by  the  US  DoD.  This  meaning  of  the  term  of  accreditation  is  the  one  that  is  part  of 
the  commonly  used  acronym,  VV&A,  which  stands  for  Verification,  Validation  and  Accreditation.  This 
acronym  has  a  specific  meaning  within  the  US  DoD  M&S  and  decision-maker  community,  since  it  integrates 
V&V  effort  within  their  formal  acceptance  decision  process. 

In  practice  however,  it  is  highly  incumbent  upon  the  V&V  User/Sponsor  acceptance  decision  needs  and 
complexity  of  the  M&S  system  as  to  which  parts  and  to  what  extent  V&V  should  be  conducted  in  an 
independent  manner.  Therefore,  the  GM-VV  adopts  a  sliding  scale  of  independence  for  V&V,  which  can  be 
selected  accordingly  to  match  the  V&V  needs.  The  justification  and  selection  of  a  proper  level  of 
independence  is  supported  within  GM-VV  through  the  use  of  the  V&V  argumentation  network.  Within  this 
sliding  scale  for  independent  V&V,  certification  and  accreditation  can  be  located  in  the  right  part  of  the  scale 
(Figure  4-6). 
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Figure  4-6:  Levels  of  Independent  V&V. 


4,2,8  V&V  Information  and  Knowledge  Management 

V&V  of  M&S  is  an  information  and  knowledge  intensive  effort.  In  particular,  during  the  V&V  of  large- 
scale,  distributed  or  complex  M&S  applications,  care  must  be  taken  to  preserve  or  reuse  information  and 
knowledge.  Therefore,  GM-VV  applies  the  memory  concept  on  both  the  V&V  project  and  enterprise  levels. 
A  memory  is  viewed  as  a  combination  of  an  information  and  knowledge  repository  and  a  community  of 
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practice.  The  repository  is  a  physical  place  where  information,  knowledge  objects,  and  artifacts  are  stored. 
The  community  of  practice  is  composed  of  the  people  who  interact  with  those  objects  to  learn,  understand 
context  and  make  decisions. 

The  V&V  project  memory  provides  the  means  to  manage  information  and  knowledge  produced  and  used 
during  the  life-time  of  an  individual  V&V  project.  V&V  is  often  an  iterative  and  recurrent  process  linked  to 
an  M&S  system’s  life-cycle,  hence  V&V  products  for  an  M&S  system  may  have  different  configurations. 
Therefore,  a  V&V  project  memory  may  also  retain  records  on  possible  different  V&V  product 
configurations.  The  V&V  enterprise  memory  retains  the  total  body  of  information  and  knowledge  from  past 
and  current  V&V  projects  to  sustain  and  support  the  cost-effective  execution  of  future  V&V  projects. 
Such  reusable  information  could  be,  for  example,  M&S  technology  or  domain  specific  recommended 
practices,  acceptability  criteria,  V&V  goal-claim  network  design  patterns,  V&V  tools  and  techniques, 
or  policies  and  standards.  On  a  more  strategic  level,  a  V&V  enterprise  memory  could  retain  information  and 
knowledge  on  V&V  project  costs  and  maturity  as  well. 
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Chapter  5  -  GM-VV  IMPLEMENTATION  FRAMEWORK* 

5.1  INTRODUCTION 

The  GM-VV  implementation  framework  translates  the  GM-VV  basie  eoneepts  of  Chapter  4  into  a  set  of 
generie  V&V  building  bloeks  (i.e.  components).  These  may  be  used  to  develop  a  tailored  V&V  solution  that 
fits  the  V&V  needs  of  any  particular  M&S  organization,  project,  application,  and  technology  or  problem 
domain.  The  implementation  framework  has  three  interrelated  dimensions:  product,  process  and  organization 
(Figure  5-1).  The  underlying  principle  of  this  framework  is  that  the  V&V  needs  of  the  V&V  User/Sponsor  in 
the  M&S  four-world  view  are  addressed  by  one  or  more  V&V  products,  those  being  the  V&V  report  and 
possibly  other  custom  V&V  products  the  V&V  User/Sponsor  may  need.  These  V&V  products  in  general 
require  intermediate  products  (i.e.  information  artifacts)  and  associated  processes  to  produce  them.  The  V&V 
processes  are  executed  by  a  corresponding  V&V  organization  that  is  responsible  for  the  development  and 
delivery  of  the  V&V  products.  In  general  the  V&V  effort  should  result  in  a  V&V  report  to  be  delivered  to 
the  customer  containing  one  or  more  of  the  information  artifacts.  Individual  needs  will  drive  which  V&V 
products  are  required. 


Figure  5-1:  GM-W  Implementation  Framework  Dimensions. 

As  indicated  in  Figure  5-1,  the  GM-VV  implementation  framework  consists  of  three  key  dimensions: 

•  Products:  The  information  artifacts  that  may  be  delivered,  developed  or  used  throughout  a  V&V 
effort.  These  artifacts  can  have  multiple  instances,  representational  and  documentation  formats. 
Examples  of  such  tailored  documentation  formats  are  the  US  DoD  accreditation  plan  and  report 
templates,  and  the  V&V  plan  and  report  templates. 

•  Processes:  The  set  of  activities  and  tasks  that  comprise  V&V  execution  as  well  as  those  management 
tasks  that  increase  the  efficiency  and  effectiveness  of  the  V&V  effort.  These  activities  and  tasks  are 
inspired  by  the  IEEE  standard  system  life-cycle  processes  model  [6]  and  can  be  carried  out 
recursively,  concurrently,  and  iteratively. 

•  Organization:  The  roles  played  either  by  people  or  by  organizations  in  the  V&V  effort.  The  roles 
are  defined  in  terms  of  responsibilities  and  obligations.  Depending  on  the  M&S  organization, 
project  and  application  domain  needs;  several  roles  could  be  played  by  separate  organizations, 
separate  people  in  one  organization  or  by  a  single  person. 


’  Parts  of  this  chapter  are  taken  from  SISO-GUIDE-001.1-2012,  “GM-W  Vol.  1:  Introduction  and  Overview”  (Copyright  2012 
by  Simulation  Interoperability  Standards  Organization  -  SISO)  [1]. 
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The  V&V  effort  culminates  in  a  V&V  report  that  is  comprised  of  the  information  generated  throughout  the 
execution  of  the  V&V  and  acceptance  decision-support  process.  The  following  sub-sections  provide  an 
overview  of  the  information  artifacts,  activities  and  roles  that  are  implemented  or  produced  during  this 
execution.  They  are  ordered  according  to  the  GM-VV  technical,  project  and  enterprise  levels.  In  this  volume 
only  high  level  descriptions  are  provided.  Detailed  descriptions  of  all  components  can  be  found  in  Vol.  2  [2]. 

It  is  important  to  re-emphasize  the  tailorable  nature  of  the  methodology.  GM-VV  provides  all  the  elementary 
information  artifacts,  activities,  tasks  and  roles  to  address  the  most  common  technical,  project  and  enterprise 
level  aspects  of  a  V&V  effort.  Depending  on  the  M&S  project  and  organizational  needs  one  could  choose 
not  to  implement  all  GM-W  components  or  one  could  choose  to  adjust  them  accordingly.  This  is  particularly 
relevant  for  M&S  organizations  that  already  have  some  project  and  enterprise  level  components  in  place, 
and  only  require  technical  level  V&V  (intermediate)  products,  processes  and  roles  to  conduct  their  V&V 
effort. 


5.2  TECHNICAL  LEVEL  GM-VV  INFORMATION  ARTIFACTS,  ACTIVITIES 
AND  ROLES 

This  sub-section  describes  the  technical  activities  performed,  the  artifacts  produced  and  the  roles  that  are 

filled  during  the  execution  of  the  V&V  effort.  These  lists  are  not  all  inclusive  and  may  be  tailored  to  reflect 

the  needs  and  constraints  of  a  specific  V&V  project. 

5,2,1  Technical  Level  Information  Artifacts 

•  V&V  Requirements:  Requirements  placed  on  the  V&V  project  deliverables  and  execution,  including 
constraints.  Note  these  are  not  the  M&S  requirements  for  the  M&S  system. 

•  V&V  Context  Information:  M&S  information  needed  prior  to  or  during  the  V&V  project.  It  captures 
information  regarding  the  M&S  problem  solving  life-cycle  and  process  such  as  the  M&S  system 
requirements,  intended  use  and  risks. 

•  V&V  Plan:  Specifies  the  V&V  execution  process,  tasks  and  experimental  frame  to  be  implemented  as 
well  as  the  associated  resources. 

•  V&V  Experimental  Frame:  A  set  of  experiments,  tests  and  conditions  used  to  observe  and  experiment 
with  the  M&S  system  to  obtain  V&V  results. 

•  V&V  Results:  The  collection  of  data  items  produced  by  applying  a  V&V  experimental  frame  to  an 
M&S  system. 

•  V&V  Argumentation  Structure:  Captures  the  derivation  of  acceptability  criteria  from  the  acceptance 
goal,  and  the  derivation  of  the  V&V  experimental  frame  specification  from  the  acceptability  criteria. 
It  provides  the  rationale  for  these  derivations.  It  integrates  the  V&V  results  into  items  of  evidence, 
and  provides  argumentation  for  the  acceptability  claims  underlying  the  acceptance  recommendation. 
(Possible  implementations  could  be  a  V&V  goal-claim  network  or  a  traceability  matrix  [9].) 

•  Acceptance  Recommendation:  An  account  or  record  containing  the  recommendations  on  the 
acceptability  of  the  M&S  system  for  the  intended  use.  This  acceptance  recommendation  integrates 
descriptions  of  all  the  information  artifacts. 

•  V&V  Report:  Accumulates  and  documents  the  information  generated  throughout  the  V&V  effort, 
along  with  information  on  how  the  V&V  effort  has  been  performed. 
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5.2.2  Technical  Level  Processes 

•  V&V  Requirements  Definition:  Defines  the  V&V  requirements  and  the  associated  V&V  context 
information  for  the  V&V  project  based  on  the  V&V  User/Sponsor  needs. 

•  Acceptance  Planning:  Transforms  the  V&V  requirements  and  context  information  into  associated 
acceptability  criteria  for  the  M&S  system. 

•  V&V  Planning:  Transforms  the  acceptability  criteria  into  the  V&V  Experimental  Frame  specification 
and  the  V&V  plan. 

•  V&V  Execution:  Implements  and  executes  the  V&V  Experimental  Frame  according  to  the  V&V  plan 
to  produce  V&V  Results;  integrates  them  into  items  of  evidence  for  the  M&S  system.  This  process  can 
include  the  following  activities: 

•  Verify  M&S  requirements; 

•  Collect,  analyze  and  apply  relevant  M&S  system  historical  information; 

•  Verify  and  validate  the  conceptual  model; 

•  Perform  verification  on  the  M&S  system  design  and/or  implementation; 

•  Verify  and  validate  the  data  and  knowledge  sets;  and 

•  Validate  the  M&S  results. 

•  V&V  Assessment  and  Integration:  Assesses  and  integrates  the  items  of  evidence  into  acceptability 
claims  regarding  whether  or  not  the  M&S  system  satisfies  the  acceptability  criteria. 

•  Acceptance  Assessment  and  Integration:  Assesses  and  integrates  the  acceptability  claims  into 
claims  regarding  to  what  extent  the  M&S  system  is  acceptable  for  the  intended  use  (i.e.  acceptance 
recommendations). 

•  V&V  Product  Delivery:  Packaging  the  information  artifacts  into  the  V&V  Report  and  delivering  it  to 
the  V&V  User/Sponsor,  and  archiving  the  information  artifacts  in  appropriate  repositories. 

5.2.3  Technical  Level  Roles 

•  Acceptance  Leader:  Responsible  for  specifying  the  acceptability  criteria,  assessing  the  acceptability 
claims  and  constructing  the  acceptance  recommendations. 

•  V&V  Leader:  Responsible  for  developing  the  V&V  plan,  assessing  and  integrating  the  V&V  results 
into  items  of  evidence,  and  constructing  the  acceptability  claims. 

•  V&V  Implementer:  Responsible  for  implementing  the  V&V  experimental  frame  and  generating  V&V 
results.  Examples  of  V&V  implementers  are  SMEs,  M&S  developers  and  test  engineers. 


5,3  PROJECT  LEVEL  GM-VV  INFORMATION  ARTIFACTS,  ACTIVITIES 
AND  ROLES 

This  sub-section  describes  the  managerial  activities  performed  at  project  level,  the  artifacts  produced  and  the 
accompanying  organizational  structure.  The  project  level  provides  a  supporting  environment  that  can 
enhance  the  effectiveness  and  efficiency  of  the  technical  V&V  work.  This  includes  a  V&V  project  memory 
that  facilitates  the  management  and  maintenance  of  the  total  body  of  V&V  information  artifacts  produced 
during  a  V&V  project.  These  lists  are  not  all  inclusive  and  may  be  tailored  to  reflect  the  needs  and 
constraints  of  a  specific  V&V  project. 
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5.3.1  Project  Level  Information  Artifacts 

•  V&V  Project  Plan:  A  coherent  arrangement  of  activities  and  tasks  to  guide  both  the  V&V  project 
execution  and  control.  Can  incorporate  or  reference  the  technical  level  V&V  plan. 

•  V&V  Project  Status  Report:  An  account  or  record  to  provide  information  on  the  conduct  of  the  V&V 
project,  its  status  and  issues. 

5.3.2  Project  Level  Processes 

•  Project  Planning:  Produces,  maintains  and  communicates  an  effective  V&V  project  plan. 

•  Project  Assessment  and  Control:  Reports  on  the  V&V  project  status  and  supports  V&V  project  plan 
execution  to  ensure  that  the  schedule,  costs,  deliverables  and  objectives  specified  in  a  V&V  agreement 
are  met. 

•  Decision  Management:  Provides  information  to  determine  the  most  beneficial  course  of  action  for  the 
V&V  project  where  alternatives  exist. 

•  Risk  Management:  Provides  information  to  identify,  analyze,  monitor  and  manage  V&V  project  risks 
continuously. 

•  Configuration  Management:  Defines  the  mechanism  to  establish  and  maintain  the  integrity  of  all 
project  deliverables,  associated  intermediate  products,  and  information  during  the  V&V  project 
execution. 

•  Information  Management:  Supports  appropriate  information  exchange  among  all  parties  and  roles 
involved  in  the  V&V  project  execution. 

•  Measurement:  Collects,  analyzes,  and  reports  data  related  to  the  overall  V&V  project,  its  performance 
and  the  quality  of  its  deliverables. 

5.3.3  Project  Level  Role 

•  V&V  Project  Manager:  Responsible  for  managing  the  V&V  project  to  assure  that  the  V&V  report  and 
possibly  other  custom  V&V  product(s)  are  developed  and  delivered  according  to  the  V&V  agreement. 


5,4  ENTERPRISE  LEVEL  GM-VV  INFORMATION  ARTIFACTS,  ACTIVITIES 
AND  ROLES 

This  subsection  describes  the  managerial  activities  performed  at  the  enterprise  level,  the  artifacts  produced 
and  the  accompanying  organizational  structure.  The  enterprise  level  provides  a  supporting  environment  that 
can  establish  a  V&V  effort  and  can  enhance  its  effectiveness  and  efficiency.  This  includes  a  V&V  enterprise 
memory  that  facilitates  the  management  and  maintenance  of  the  total  body  of  V&V  information  artifacts, 
knowledge  and  products  required  to  sustain  the  delivery  of  V&V  products  by  a  V&V  supplier  for  any  M&S 
project.  These  lists  are  not  all  inclusive  and  may  be  tailored  to  reflect  the  needs  and  constraints  of  a  specific 
enterprise  that  executes  V&V  projects. 

5,4,1  Enterprise  Level  Information  Artifacts 

•  V&V  Agreement:  A  contract,  statement  of  work  or  any  type  of  agreement  between  a  V&V  client  entity 
and  a  V&V  supplier  entity  for  the  delivery  of  a  V&V  product(s). 
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5.4.2  Enterprise  Level  Processes 

•  Agreement  Management:  Establishes  and  manages  the  V&V  agreement  between  V&V  client  and  the 
supplier  entity. 

•  Life-Cycle  Model  Management:  Defines,  maintains  and  ensures  availability  of  V&V  life-cycle  models 
suitable  for  carrying  out  any  V&V  project. 

•  Project  Portfolio  Management:  Initiates  and  sustains  necessary,  sufficient  and  suitable  V&V  projects 
in  order  to  meet  the  strategic  V&V  supplier  entity  objectives. 

•  Resource  Management:  Ensures  that  necessary  resources  are  provided  for  carrying  out  V&V  projects 
and  that  skills,  competencies,  and  infrastructure  are  maintained,  consistent  with  the  enterprise  entity 
needs. 

•  Quality  Management:  Ensures  that  the  delivered  V&V  product(s)  meets  the  enterprise  entity  quality 
standards  and  achieves  V&V  User/Sponsor  satisfaction. 

5.4.3  Enterprise  Level  Roles 

•  V&V  Enterprise  Manager:  Responsible  for  managing  the  environment  in  which  V&V  projects  are 
conducted.  This  role  contributes  to  the  arrangement  of  a  V&V  agreement  from  the  supplier  side. 

•  V&V  User/Sponsor:  Responsible  for  specifying  the  V&V  requirements  and  endorsing  the  delivered 
V&V  product(s).  This  role  contributes  to  the  arrangement  of  a  V&V  agreement  from  the  client  side. 
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6.1  OVERVIEW 

The  GM-VV  tailoring  framework  applies  four  basie  tailoring  approaches: 

•  Tailoring  by  Extension:  Adaptation  of  the  implementation  framework  by  adding  custom  V&V 
products,  processes,  activities,  tasks  and  roles.  For  example,  a  V&V  Client  organization  or 
application  domain  may  require  additional  custom  artifacts  not  foreseen  by  the  GM-VV. 

•  Tailoring  by  Reduction:  Adaptation  of  the  implementation  framework  by  deleting  products, 
processes,  activities,  tasks  and  roles  due  to  constraints  such  as  inaccessibility  of  data  and 
information  protected  by  intellectual  property  rights,  security  or  technical  restrictions. 

•  Tailoring  by  Specialization:  Adaptation  of  the  implementation  framework  by  modifying  domain 
specific  V&V  methods,  techniques  and  data  that  are  unique  for  a  V&V  project,  organization  or 
application. 

•  Tailoring  by  Balancing:  Adaptation  of  the  implementation  framework  by  fitting  a  suitable 
cost-benefit-ratio  towards  an  acceptance  recommendation.  The  level  of  acceptable  M&S  use  risk 
should  drive  the  rigor  and  resources  employed  for  V&V.  Therefore,  in  this  approach  one  tries  to 
balance  aspects  such  as: 

•  M&S  use-risk  tolerances  and  thresholds; 

•  Criticality  and  scope  of  the  acceptance  decision; 

•  Scale  and  complexity  of  the  M&S  system;  and 

•  Information  security,  with  V&V  project  resource  variables  (e.g.  time  schedule,  budget, 
V&V  personnel  skills  and  infrastructure). 

The  GM-VV  implementation  framework  provides  a  set  of  generic  reusable  components  for  each  of  the 
categories  depicted  in  Figure  6-1.  These  components  are  grouped  into  three  interrelated  organizational  levels 
(i.e.  enterprise,  project  and  technical)  where  V&V  of  M&S. 


'  Parts  of  this  chapter  are  taken  from  SISO-GUIDE-001.1-2012,  “GM-W  Vol.l:  Introduction  and  Overview”  (Copyright  2012 
by  Simulation  Interoperability  Standards  Organization  -  SISO)  [1]  and  “GM-W  Vol.  2:  Implementation  Guide”  [2]. 
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Figure  6-1:  GM-W  impiementation  and  Taiioring  Framework  Appiication  Overview. 


The  technical  level  comprises  a  set  of  technical  components  that  together  constitute  a  generic  engineering 
life-cycle  template  for  structuring  the  technical  V&V  work  necessary  to  develop  and  deliver  an  acceptance 
recommendation  (i.e.  the  technical  activities  performed,  the  information  artifacts  produced  and  the  roles 
fulfilled).  To  develop  a  quality  (e.g.  timely,  accurate  and  relevant)  acceptance  recommendation,  the  technical 
V&V  work  should  be  executed  in  a  well-controlled  and  organized  manner.  Therefore,  the  GM-VV 
recommends  performing  the  technical  V&V  work  as  part  of  a  managed  project.  The  project  level  of  the 
GM-VV  implementation  framework  provides  a  set  of  project-oriented  components  that  together  constitute  a 
generic  project  structure  template  for  organizing  and  managing  the  technical  V&V  work.  When  a  V&V 
supplier  establishes,  directs  and  supports  the  execution  of  multiple  V&V  projects  and  delivery  of  V&V 
products,  the  GM-VV  recommends  establishing  a  permanent  V&V  business  environment.  Such  a  permanent 
V&V  organization  helps  to  improve  the  quality,  reduce  costs  and  lead  time  of  these  V&V  projects  and 
products.  The  enterprise  level  of  the  GM-VV  implementation  framework  provides  a  set  of  enterprise- 
oriented  components  that  together  constitute  a  generic  enterprise  level  organization  (i.e.  a  line  organization) 
template  for  establishing  and  operating  a  permanent  V&V  organization. 

NOTE:  The  GM-VV  implementation  framework  solely  focuses  on  the  V&V  supplier  perspective.  The  V&V 
client  and  its  V&V  User/Sponsor  role  are  only  identified  to  describe  the  involvement  in  and  interaction  with 
the  V&V  supplier. 

The  purpose  of  the  GM-VV  tailoring  framework  is  to  customize  the  GM-VV  implementation  framework 
components  (i.e.  products,  processes,  organizational  roles)  to  satisfy  the  specific  requirements  and 
constraints  of: 

•  An  organization  that  is  employing  the  GM-VV  (e.g.  company  policies,  standards); 

•  A  domain  in  which  the  GM-VV  is  employed  (e.g.  standards,  regulations,  technologies); 
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•  A  M&S/V&V  supplier  delivering  V&V  products  or  services  (e.g.  standards,  processes);  and 

•  A  M&S/V&V  project  (e.g.  time,  budget,  scale,  complexity,  risk,  resources). 

This  tailoring  is  accomplished  in  two  phases.  In  the  first  phase  of  the  GM-VV  tailoring  framework, 
the  implementation  framework  components  are  utilized  to  establish  concrete  V&V  solution  instances  on  one 
or  more  of  the  three  organizational  levels  (i.e.  a  V&V  enterprise,  V&V  project  or  technical  V&V  approach). 
The  GM-VV  recognizes  that  a  particular  M&S  organization,  project,  technology  or  problem  domain  may  not 
need  all  three  organizational  levels  or  all  components  nor  even  use  them  directly  as-is.  Therefore, 
the  GM-VV  implementation  framework  organizational  levels  and  components  are  selected,  combined  and 
modified  accordingly,  to  obtain  a  tailored  V&V  solution.  For  instance  an  M&S  organization  may  already 
have  an  M&S  project  and  enterprise  level  in  place,  and  only  require  technical  level  V&V  (intermediate) 
products,  processes  and  roles  to  conduct  their  technical  V&V  work.  Successful  application  of  the  first  phase 
of  the  tailoring  framework  results  in  a  modified  or  new  V&V  solution  instance  conforming  to  the  GM-VV 
architectural  templates  (i.e.  in  a  structure  and  organizational  manner).  The  aforementioned  tailoring 
approaches  should  be  used  for  this  purpose:  extension,  reduction,  specialization  and  balancing  [1]. 

In  the  second  phase  these  same  tailoring  approaches  are  applied  throughout  the  operational  life-time 
(i.e.  permanent  organization  or  project)  or  execution  (i.e.  technical  approach)  of  each  V&V  solution  instance. 
This  type  of  tailoring  comprises  run-time  optimization  of  the  instantiated  V&V  processes  at  all  three 
organizational  levels.  At  a  technical  level  this  could  imply  the  application  of  a  risk-based  V&V  approach  to 
prioritize  the  acceptability  criteria,  allocate  specific  V&V  techniques  and  tools  based  on  V&V  User/Sponsor 
risk  tolerance  levels.  On  the  project  level  this  could  be  the  alignment  of  technical  V&V  activities  with  the 
progress  of  the  M&S  system’s  life-cycle  phases,  and  balancing  the  available  V&V  resources  over  each  M&S 
life-cycle  phase  or  (work)  products.  On  the  enterprise  level  this  could  mean  balancing  the  cost-risk  of  new 
investments  in  training  of  personnel  or  V&V  tool  infrastructure  development  against  a  future  V&V  project 
order  intake  volume. 

6.2  TAILORING  CONSIDERATIONS  FOR  INSTANTIATING  THE 
TECHNICAL  LEVEL 

The  technical-level  components  of  the  GM-VV  implementation  framework  are  typically  the  components  that 
all  users  of  the  GM-VV  should  instantiate.  This  technical  level  provides  a  generic  template  for  a  structured 
execution  of  technical  V&V  activities  and  tasks.  In  this  template  technical-oriented  V&V  practices,  tools  and 
techniques  native  to  each  individual  M&S  organization,  project,  and  technology  or  application  domain 
should  be  integrated.  Among  others  such  technical-oriented  V&V  practices,  tools  and  techniques  include: 

•  Informal,  formal,  static,  and  dynamic  V&V  techniques  to  be  used; 

•  Risk  analysis  techniques  and  methods; 

•  Argumentation  structure  formats,  techniques  and  methods; 

•  Referent  development  techniques  and  methods; 

•  Domain  specific  or  standard  acceptability  criteria; 

•  Domain  specific  or  standard  V&V  documentation  templates;  and 

•  Experimental  design  techniques  and  methods  (DOE),  Information  management  and  configuration 
tools. 

A  more  detailed  discussion  of  such  technical  V&V  practices,  tools  and  techniques  is  beyond  the  scope  of  this 
document.  The  interested  reader  can  find  references  on  such  technical-oriented  V&V  practices,  tools  and 
techniques  in  GM-VV  Vol.  3  [3].  The  types  of  technically-oriented  practices,  tools  and  techniques  selected 
and  when  they  should  be  applied  during  the  V&V  of  an  M&S  system,  is  an  outcome  of  the  run-time 
execution  of  a  technical  V&V  approach. 
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Such  a  technical  V&V  approach  is  thus  a  tailored  instance  of  the  generic  life-cycle  for  structuring  the 
technical  V&V  work  of  the  GM-VV  implementation  framework.  A  technical  V&V  approach  describes 
which  technical  V&V  activities  and  tasks  are  executed  and  how  they  map  onto  the  M&S  system  life -cycle 
phases  or  (work)  products.  It  typically  scopes  the  V&V  technical  work,  structures  the  technical  V&V 
activities  and  tasks  to  be  performed  in  a  logical  order.  Furthermore  it  identifies  the  technical  standards  to  be 
used,  the  V&V  design  techniques  to  be  applied  and  the  associated  completion  criteria.  The  technical  V&V 
approach  depends  strongly  on  the  M&S  application  and  problem  domain,  the  M&S  organization  and  project 
context  in  which  the  V&V  is  executed,  and  more  importantly  on  the  V&V  needs  of  the  V&V  client. 
This  means  that  the  GM-VV  technical  level  components  must  be  tailored  by  specialization,  reduction  and 
extension  to  gain  a  suitable  and  matching  technical  V&V  approach  for  the  M&S  system  under  consideration. 

A  structured  V&V  engineering  life -cycle  is  obtained  if  the  technical  V&V  approach  has  been  instantiated 
correctly.  This  helps  to  assure  that  during  the  execution  of  the  V&V  work;  the  decisions,  actions  and 
information  will  be  traceable,  reproducible,  transparent  and  documented.  This  is  accomplished  by  a  tailored 
(e.g.  by  specialization  and  extension)  implementation  of  the  V&V  Argumentation  Structure  which  is  the 
practical  implementation  of  the  GM-VV  evidence  based  and  structured  reasoning  concept  introduced  in 
GM-VV  Vol.  1  [1].  How  such  decisions  are  made  during  the  execution  of  this  technical  V&V  approach 
depends  on  the  V&V  project  context  and  involves  the  GM-VV  tailoring  approaches,  in  particular  balancing. 

Balancing  approaches  are  needed  during  technical  V&V  activities  since  it  is  practically  impossible  to  verify 
or  validate  an  entire  M&S  system  in  any  project.  Exhaustive  verification  and  validation  (i.e.  100%  coverage 
of  all  aspects)  of  an  M&S  system  exists  only  in  theory  [18];  requiring  infinite  time  and  V&V  resources. 
In  practice,  there  is  always  a  limited  time  and  budget  available  for  a  V&V  project.  Moreover,  there  is  always 
the  pressure  on  the  M&S  system  development  to  provide  the  needed  capabilities  (i.e.  functionality)  on  time 
and  usually  more  capabilities  (i.e.  nice-to-have  features).  In  practice  this  means  that  the  original  time  and 
budget  allocated  for  performing  V&V  is  often  reduced  by  such  M&S  system  development  requests  and 
constraints.  This  requires  continuously  balancing  the  time  schedule,  budget  and  resources  available  for  V&V 
against  what  should  and  could  be  verified  or  validated  throughout  the  life-time  of  a  V&V  project. 

Risk-based  techniques  and  methods  are  practical  means  of  balancing.  Risk-based  V&V  centers  the 
verification  and  validation  around  the  M&S  use  risks.  M&S  use  risks  are  the  risks  directly  related  to  usage  of 
the  M&S  system  and  what  the  impact  could  be  if  the  M&S  system  isn’t  (completely)  fit  for  the  intended  use. 
Risk-based  V&V  identifies  and  analyzes  the  M&S  use  risks,  and  aims  at  addressing  these  risks  by  guiding 
the  technical  V&V  activities  towards  the  level  of  risk  of  each  identified  risk  item. 

A  risk-based  approach  responds  to  these  M&S  use  risks  as  follows: 

•  Target  Technical  V&V  Activities:  Allocating  V&V  effort  and  selecting  V&V  techniques  based  on 
the  level  of  risk  of  each  identified  risk  item;  matching  the  rigor  and  extensiveness  of  V&V 
techniques  to  the  level  of  risks. 

•  Sequencing  of  Technical  V&V  Activities:  Prioritizing  the  risk  items,  starting  with  verifying  and 
validating  the  most  important  M&S  use  risk  items  first  and  work  down  to  the  less  important  ones. 

•  Reduction  of  Technical  V&V  Activities:  If  the  initial  time,  budget  and  resources  are  limited  or  are 
reduced  throughout  the  life-time  of  the  V&V  project,  V&V  activities  and  techniques  can  be  reduced 
in  reverse-risk  priority  order,  starting  with  least  important  ones. 

•  Reporting  of  Technical  V&V  Results:  Reporting  V&V  results  in  terms  of  residual  M&S  use  risks 
(e.g.  V&V  solutions  executed,  not  executed,  executed  with  limitations  or  omitted). 

When  applying  a  risk-based  approach,  V&V  project  managers  should  ensure  that  the  risk-based  V&V 
activities  and  techniques  recommended  by  the  M&S  use-risk  assessment  corresponds  to  the  overall  V&V 
project  organization  and  plan. 
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Risk-based  approaches  have  proven  to  be  very  effective  for  V&V  of  software,  hardware  and  M&S  systems 
alike,  and  are  therefore  recommended  by  GM-VV  [34], [35], [36].  However,  it  is  beyond  the  scope  of  this 
guidance  document  to  give  a  complete  overview  of  risk-based  techniques  and  methods.  The  interested  reader 
is  referred  to  the  GM-VV  Vol.  3  for  references  on  this  topic  [3].  It  must  also  be  noted  that  a  pure  risk-based 
approach  to  V&V  can  leave  blind  spots  [34]. 


6.3  TAILORING  CONSIDERATIONS  FOR  INSTANTIATING  THE  PROJECT 
LEVEL 

The  GM-VV  concept  of  a  V&V  project  can  be  viewed  as  a  unique  process  comprised  of  coordinated  and 
controlled  activities  that  address:  technical  V&V  work  planning  in  terms  of  cost,  timescales  and  milestones; 
measuring  and  checking  progress  against  this  planning;  and  selecting  and  taking  corrective  actions  when 
needed  [1].  The  project  level  of  the  GM-VV  implementation  framework  provides  the  components  to 
implement  this  V&V  project  concept.  It  is  inevitable  that  the  project-level  components  provided  by  GM-VV 
do  not  directly  match  the  way  V&V  is  organized  and  managed  in  specific  M&S  organization  or  project. 
Nevertheless,  these  aspects  are  important  for  assuring  the  quality  (e.g.  timely,  accurate  and  relevant)  of  the 
V&V  products,  and  thus  such  aspects  should  be  considered  and  addressed  by  all  V&V  suppliers; 
independent  of  whether  the  V&V  is  performed  by  an  external  organization,  a  separate  business  unit  in  the 
M&S  organization  or  solely  from  within  the  M&S  development  project.  Therefore,  the  concept  of  a 
managed  V&V  project  can  be  instantiated  by  tailoring  the  GM-VV  project-level  components  as  either  a 
separate  V&V  project  or  as  a  sub-project  or  work  package  of  a  larger  M&S  project.  This  constitutes  a 
generic  organizational  scheme  to  organize  and  manage  the  technical  V&V  work. 

Independent  V&V  (IV&V)  requires  V&V  projects  that  have  the  highest  level  of  independence  (see  GM-VV 
Vol.  1  for  more  details  on  IV&V  [1]).  In  that  case  the  V&V  project  organization  and  team  are  fully  separated 
from  the  M&S  development  project.  Such  V&V  projects  are  executed  by  a  V&V  supplier  outside  the  own 
M&S  organization.  A  less  strict  level  of  independence  can  be  achieved  by  a  having  a  dedicated  V&V 
supplier  organization  unit  inside  the  own  M&S  organization.  In  this  case  the  V&V  project  manager  and  his 
or  her  team  have  a  V&V  project  budget  which  is  separate  from  the  M&S  development  budget,  and  reports  to 
the  higher  enterprise  management  of  the  own  M&S  organization.  In  the  case  of  a  V&V  sub-project,  the 
V&V  project  manager  and  team  usually  work  on  the  same  level  with  the  M&S  development  project  manager 
and  team  under  the  direction  of  an  overall  M&S  program/project  manager.  When  the  V&V  is  executed  as  a 
V&V  work  package  in  M&S  project,  the  V&V  team  is  usually  integrated  within  the  M&S  development 
project,  and  there  is  no  separate  V&V  project  manager  or  project  plan.  The  V&V  project  manager  role  is 
assumed  by  the  M&S  project  manager  itself  In  that  case  there  is  no  independence. 

Whether  the  V&V  work  should  be  executed  as  a  separate  project,  a  sub-project  or  a  work  package  depends 
on  the  V&V  client  organization,  what  level  of  independence  the  V&V  client  requires  and  the  scale  of  the 
M&S  project.  For  instance,  if  the  M&S  supplier  organization  is  the  V&V  client,  conducts  V&V  on  various 
projects  and  has  high  quality  standards  (e.g.  for  customer  satisfaction  and  marketing  perspective),  a  separate 
V&V  project  is  recommended  for  the  technical  V&V  work.  In  cases  where  customers  of  M&S  systems  do 
not  require  a  high-level  of  V&V  independence,  a  V&V  sub-project  or  work-package  within  a  larger  M&S 
project  is  recommended.  When  a  dedicated  V&V  organization  is  contracted,  a  separate  V&V  project  or  sub- 
project  within  a  larger  M&S  project  is  the  most  obvious  option.  A  separate  V&V  (sub)  project  is  in  particular 
the  best  option  when  the  V&V  client  is  a  different  organization  than  the  M&S  supplier,  and  wants  a  fully 
independent  V&V  of  the  M&S  system  it  acquires.  All  are  forms  of  tailoring  the  V&V  project  level  by 
specialization. 

For  large,  complex  or  safety-critical  M&S  projects,  usually  multiple  levels  of  V&V  independence  are 
required  for  the  M&S  system,  meaning  that  the  V&V  team  is  a  mix  of  permanent,  temporary,  internal  and 
external  personnel.  Hence,  having  a  separate  V&V  project,  with  a  separate  V&V  manager  and  project  plan 
from  the  M&S  development  project  is  then  highly  recommended  to  organize  and  manage  the  technical  work 
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properly.  Good  alignment,  cooperation  and  communication  should  be  maintained  between  both  the  V&V 
and  M&S  project  to  assure  the  right  quality  (i.e.  fit  for  purpose)  M&S  system  is  delivered.  In  smaller  M&S 
organizations  or  projects,  where  everybody  contributes  to  every  activity  it  is  harder  to  differentiate  the 
technical  V&V  work  and  roles  from  those  of  the  M&S  development.  Hence,  it  is  easier  and  cheaper  to 
embed  V&V  as  a  work  package  inside  the  M&S  project.  In  general,  higher  levels  of  independence  or 
outsourcing  to  external  V&V  organizations  comes  with  higher  organizational  (i.e.  managerial)  effort  and 
costs  but  on  the  other  hand  when  done  correctly  provide  more  effective  V&V.  Therefore,  the  decision  to 
setup  and  manage  the  technical  V&V  work  as  a  separate  project,  sub-project  or  work  packages  should 
involve  a  careful  analysis  of  the  balance  between  aspects  such  as  the  project  risks,  scale  of  the  M&S  project 
or  organization,  cost,  time  and  other  resources  required.  This  is  a  form  of  tailoring  by  balancing. 

For  Post-hoc  V&V  projects  the  V&V  is  conducted  in  retrospect  on  an  M&S  legacy  system  after  development 
or  application.  This  is  not  the  most  efficient  form  of  V&V.  Post-hoc  V&V  projects  are  often  seen  within 
organizations  that  reuse  or  acquire  M&S  systems  (e.g.  modified-off-the -shelf,  commercial-off-the-shelf) 
from  an  external  M&S  supplier.  In  such  case  the  acquiring  organization  usually  wants  an  independent  V&V 
to  assure  that  the  reused  or  acquired  M&S  system  will  fit  the  intended-use.  V&V  is  then  best  executed  as  a 
separate  V&V  project  by  an  external  third-party  V&V  supplier.  Since  in  this  case  the  M&S  supplier  and  the 
V&V  supplier  are  separate  entities,  some  alignment,  cooperation  and  communication  between  them  should 
be  established.  This  is  to  ensure  that  the  V&V  supplier  can  access  the  M&S  system  itself  and  associated 
information  (e.g.  conceptual  model,  design  specifications  and  test  data),  and  address  any  M&S  supplier 
intellectual  property  rights  and  security  issues  that  may  apply. 

It  must  be  stressed  that  the  GM-VV  project-level  organizational  pattern  and  components  are  not  intended  as 
a  substitute  for  standard  project  management  and  organization  practices;  instead  they  contain  complementary 
V&V  project-level  aspects  that  should  be  used  in  conjunction  with  standard  practices  [SlSO-REF-039-2013]. 
Therefore,  the  GM-VV  project-level  components  are  not  all  inclusive  and  should  be  tailored  to  reflect  the 
needs,  objectives  and  constraints  of  an  M&S  project  or  M&SA^&V  organization.  For  example  in  the  case 
where  the  V&V  effort  is  established  as  a  standalone  V&V  project,  all  GM-VV  project-level  components 
may  have  to  be  implemented  from  scratch  in  order  to  organize  and  manage  the  V&V  project.  An  M&S 
organization  that  already  has  similar  processes  in  place,  may  only  adapt  these  to  meet  the  specific 
managerial  needs  of  the  M&S  project.  These  are  forms  of  tailoring  the  GM-VV  by  reduction,  extension  and 
specialization. 

A  prerequisite  for  instantiating  the  project-level  components  and  successfully  executing  a  V&V  project  is 
that  technical  V&V  work  is  also  executed  in  a  structured  manner.  If  no  such  approaches  exist  yet  within  the 
M&S  project  or  organization,  this  should  be  first  developed.  For  developing  a  structured  technical  V&V 
approach,  instantiation  of  the  GM-VV  technical-level  components  is  recommended. 


6.4  TAILORING  CONSIDERATIONS  FOR  INSTANTIATING  THE 
ENTERPRISE  LEVEL 

A  permanent  organization  for  supplying  V&V  services  and  products  can  be  implemented  as  an  autonomous 
company  or  as  an  organizational  unit  part  of  a  larger  company.  The  first  type  of  V&V  suppliers  are 
companies  who  have  as  their  core  business  the  delivery  of  V&V  products  (e.g.  V&V  reports,  services, 
expertise  and  tools)  to  M&S  developer,  user  or  regulation  organizations  (i.e.  external  V&V  clients). 
The  latter  types  of  V&V  suppliers  are  M&S  developer,  user  or  regulation  organizations  that  have  their  own 
internal  V&V  organization  unit  to  support  their  own  M&S  projects  (i.e.  internal  V&V  clients);  and  possibly 
also  as  an  additional  business  for  external  V&V  clients. 

It  is  not  necessary  for  all  V&V  suppliers  to  establish  a  permanent  V&V  organization.  V&V  products  can  be 
delivered  using  project-based  approach  on  a  case  by  case  basis.  However,  if  the  V&V  supplier  executes 
V&V  projects  and  delivers  V&V  products  on  a  regular  basis  to  one  or  more  V&V  clients  it  can  become 
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more  cost-effective  to  set  up  a  permanent  organization  for  V&V.  A  V&V  supplier  should  consider  this 
option  when  there  is: 

•  Increasing  V&V  efforts  and  costs; 

•  Quality  reduction  in  V&V  projects  and  products; 

•  Lack  of  internal  V&V  standards,  policies  and  guidance; 

•  Lack  of  internal  coordination  of  V&V  projects  and  products; 

•  Insufficient  reuse  of  prior  knowledge,  tools,  techniques,  facilities  and  lessons-leamed; 

•  Lack  of  experienced  V&V  personnel  or  reduction  of  their  knowledge  and  skills; 

•  Insufficient  means  to  enhance  V&V  project  and  product  quality; 

•  Confusion  regarding  V&V  project  responsibilities;  and 

•  Lack  of  V&V  assessment  objectivity  and  independence. 

To  determine  if  a  permanent  V&V  organization  is  indeed  worth  the  investment  requires  a  cost  benefit 
analysis  between  the  resources  required  to  setup,  manage  and  maintain  a  permanent  V&V  organization, 
and  resulting  benefits  such  as  improved  V&V  quality,  cost  savings  and  lead-time  reduction. 
This  determination  must  also  consider  the  V&V  supplier  organization’s  own  objectives,  the  problem  and 
application  domain  in  which  it  operates,  and  the  V&V  clients  it  serves.  These  are  forms  of  tailoring  the 
GM-VV  by  balancing. 

One  must  remember  that  there  is  neither  a  fixed  set  of  requirements  or  rules  to  do  so  nor  an  ideal  one 
size-fit-all  blue  print  for  the  implementation  of  a  permanent  V&V  organization.  Therefore,  the  GM-VV 
enterprise-level  components  may  not  all  be  required  and  should  be  tailored  to  reflect  the  needs,  objectives 
and  constraints  of  a  specific  V&V  supplier.  For  example  in  the  case  where  a  permanent  V&V  supplier  is 
established  as  a  new  stand-alone  company,  it  may  have  to  implement  all  enterprise-level  components  from 
scratch.  For  an  existing  company  that  wants  to  establish  permanent  V&V  supplier  within  its  own 
organization  may  already  have  similar  enterprise  products,  processes  and  organizational  roles  in  place  and 
may  only  adapt  these  to  meet  the  specific  needs  of  this  internal  V&V  unit.  These  are  forms  of  tailoring  the 
GM-VV  by  reduction,  extension  and  specialization. 

A  prerequisite  for  instantiating  the  enterprise-level  components  and  to  successfully  sustain  a  permanent 
V&V  organization  is  that  V&V  projects  are  executed  in  a  structured  manner  on  both  project  organizational 
and  technical  level.  If  no  such  approaches  or  methods  exist  yet  within  the  V&V  supplier,  they  should  be  first 
developed.  For  developing  a  new  structured  V&V  approach  or  method  it  is  recommended  to  instantiate  the 
GM-VV  project  and  technical-level  components. 
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7.1  INTRODUCTION 

For  demonstration  and  validation  of  the  GM-VV  in  itself,  several  possible  case  studies  were  identified  and 
evaluated.  Two  case  studies  were  selected:  a  relatively  small  scoped  one  and  a  more  complex  one. 
Both  cases  were  instrumental  and  proofed  to  be  vital  for  the  on-going  development  of  the  methodology. 
The  cases  are  briefly  introduced  in  this  chapter  with  the  intention  to  provide  insight  in  the  domain  and  the 
verification  and  validation  efforts. 

All  activities  were  bounded  by,  as  in  any  real-world  case,  availability  of  resources  (time,  budget,  etc.). 
The  first  case  was  identified  as  the  “Flashing  lights”  case,  when  the  Dutch  government  instantiated  research 
experiment  of  handling  traffic  on  a  highway  at  an  incident  location.  The  second  case  was  identified  as 
“Fleavy  weather  ship  handling”.  In  this  second  case,  the  Royal  Netherlands  Navy  started  an  R&D  experiment 
to  investigate  the  effects  of  motion  on  human  performance  of  ‘officers  of  the  watch’. 

The  following  paragraphs  provide  a  short  impression  of  both  cases.  The  referred  reports  provide  more  in 
depth  details  on  the  technicalities  of  the  problem  domain  and  findings  in  applying  the  GM-VV  methodology. 
The  overall  final  conclusion  is  that  the  theoretical  model  of  GM-VV  is  actually  performing  according  to 
expectation,  but  future  work  is  needed  to  address  techniques  and  tools  to  effectively  support  further 
acceptance  by  the  community  and  implementation. 

Notice  that  the  case  studies  were  designed  with  the  primary  objective  to  test  the  GM-VV.  The  outcomes  of 
the  cases  with  regard  to  the  simulations  being  studied  as  such  must  therefore  be  seen  as  a  useful  by-product. 
However  where  possible,  case  studies  were  sought  that  provided  synergetic  effects.  This  was  achieved  in 
Case  Study  2  where  a  running  experiment  was  extended  by  a  cooperative  effort  from  the  V&V  team,  hr  this 
way,  not  only  was  the  value  of  the  Heavy  Weather  experiment  enhanced,  but  to  a  large  extent  it  also 
transformed  the  V&V  case  study  into  a  real  V&V  case. 


7.2  CASE  STUDY  I  “FLASHING  LIGHTS” 

The  “Flashing  Lights”  case  was  the  first  case  to  test-drive  the  GM-VV  methodology.  This  case  comprised  a 
recently  executed  test  and  evaluation  study  for  Dutch  Ministry  of  Transport,  Public  Works  and  Water 
Management.  For  this  study,  the  Netherlands  Organisation  for  Applied  Scientific  Research  TNO  driving 
simulator  was  used  as  a  testbed  for  the  investigation  of  the  effects  of  a  new  traffic  accident  signalling 
strategy. 

The  workshop  sessions  of  the  “Flashing  Lights”  case  were  conducted  by  a  team  consisting  of  GM-VV 
experts  from  MSG-073  countries  while  working  together  with  TNO  driving  simulator  experts  and  the  M&S 
customer  representatives.  The  findings  of  this  case  study  were  published  and  presented  in  a  Final  Report 
[42]  to  the  NMSG,  published  in  2010,  and  secondly  presented  in  a  paper  [43]  and  presentation  of  the  SISO 
Simulation  Interoperability  Workshop  held  in  Spring  2010. 

7.2.1  Real-World  Problem:  Reduction  of  Traffic  Jams 

Within  the  densely  populated  country  of  the  Netherlands,  traffic  jams  are  becoming  an  ever  more  serious 
problem  with  significant  economic  consequences.  The  Dutch  Traffic  Information  Centre  estimated  the  total 
cost  of  traffic  jams  (in  the  Netherlands)  in  2007  at  over  a  billion  Euros.  Nowadays,  traffic  jams  are 
increasing  in  length,  frequency  and  duration.  Traffic  density  in  the  Netherlands  is  so  high  that  when 
accidents  happen  on  one  of  the  main  highways,  even  other  parts  of  the  highway  infrastructure  is  severely 
implicated.  There  is  strong  need  in  the  Netherlands  for  short-term  solutions  to  reduce  traffic  jams  and 
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improve  the  efficiency  of  the  current  Dutch  high  way  system.  For  this  purpose,  the  implementing  body  of  the 
Dutch  Ministry  of  Transport,  Public  Works  and  Water  Management,  called  Rijkswaterstaat  (RWS)  initiated 
the  FileProof  research  program.  This  program  focuses  on  research  into  60  possible  counter  measures  for 
traffic  jams  that  can  be  implemented  fairly  quickly  into  the  Dutch  highway  system. 

One  of  these  possible  counter  measures  is  the  directive  “Flashing  Lights  Off  on  Accident  Site”.  Flowever, 
before  implementing  this  directive  RWS  wants  to  be  sure  that  the  directive  successfully  reduces  traffic  jams 
around  accident  sites  (both  day  and  night)  and  do  not  impose  unacceptable  risks  for  the  safety  of  rescue 
workers  on  site.  One  possible  way  to  solve  this  problem  statement  is  to  implement  a  new  directive  for 
emergency  services  and  workers  at  incident  sites.  If  these  objectives  can  be  assured  then  the  directive  is 
considered  to  be  an  acceptable  solution. 

The  question  about  the  validity  of  the  simulator  supported  experiment  results,  i.e.  M&S  results,  still 
remained  open.  A  minimal  confidence  level  is  required  on  the  validity  of  the  experiments  to  support  the 
claim  that,  whatever  the  outcome,  the  results  are  indeed  acceptable  to  make  a  valid  conclusion. 
This  depends  both  on  the  collected  evidence  of  driving  simulator  itself  and  how  the  simulator  has  been 
utilized  in  the  experimental  set-up. 

7,2,2  Product  World:  Driving  Simulator 

The  TNO  driving  simulator  is  a  six  Degree  Of  Freedom  (6DOF)  moving  base  simulator,  which  has  proven  to 
be  an  acceptable  research  tool  for  investigating  human  driver  behaviour  in  many  other  projects.  Flowever, 
the  Flashing  lights  scope  has  never  been  investigated  before,  hence  were  never  simulated  in  the  simulator. 
Therefore,  TNO  did  conduct  a  feasibility  study  to  determine  how  flashing  lights  can  be  simulated  as 
convincingly  realistic  as  possible.  The  solution  TNO  finally  implemented  comprised  a  flashing  light  model 
and  environmental  light  effects  in  the  3D  image  generator  software  of  the  visual  system.  This  visual  set-up 
has  been  combined  with  an  additional  projector  to  simulate  environmental  light  conditions  inside  the  mock- 
up  (compensating  for  screen  projection  artifacts).  The  3D  images  of  the  flashing  lights  and  the  live  mock-up 
light  conditions  were  synchronized.  Flowever  proving  the  possibility  to  simulate  flashing  lights  in  the 
existing  driving  simulator  doesn’t  directly  imply  the  simulator  to  be  acceptable  for  the  testing  and  evaluating 
the  new  flashing  lights  directive.  Acceptable  in  this  sense  means  that  the  driving  simulator  should  be  realistic 
enough  to  induce  effects  on  human  driving  behaviour  due  to  flashing  lights  that  are  similar  or  sufficiently 
similar  to  the  one  of  a  human  in  the  real-world  environment  under  the  same  conditions. 


Figure  7-1:  The  TNO  Driving  Simuiator. 


7,2,3  First  Conclusions 

As  this  experiment  was  the  first  real  attempt  to  apply  the  GM-VV  methodology,  the  findings  while  applying 
theory  in  practice  were  meticulously  recorded  and  used  for  later  evaluation.  As  a  result  of  this  exercise  the 
present  set  of  GM-VV  documents  went  through  a  major  restructuring  process  which  finally  led  to  a  change 
in  the  Product  Nomination  for  a  SISO  Standard.  This  process  also  led  to  a  reduced  document  set  as 
duplications  between  documents  were  removed,  ft  was  also  found  that  more  case  studies  were  necessary  to 
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fully  fine  tune  the  methodology  for  practical  application  in  a  wider  community  and  support  by  automated 
tools  is  a  mandatory  pre-requisite. 


7.3  CASE  STUDY  2  “HEAVY  WEATHER  SHIP  HANDLING” 

The  second  case  study  was  chosen  and  designed  around  a  mutual  understanding  on  the  value  of  a  W&A 
effort  for  the  design  and  outcomes  of  an  M&S-based  experiment  for  the  Royal  Dutch  Navy.  The  team  that 
designed  the  experiment  covered  specialists  from  the  human  factors  domain  through  simulation  engineers 
from  various  organisations  involved  and  the  VV&A  experts.  For  this  experiment  a  full  motion  research 
simulator,  called  DESDEMONA,  was  equipped  and  prepared  to  facilitate  the  conditions  required  by  the 
experiment. 

The  proposition  of  the  acceptance  goal,  the  GM-VV  wording  of  the  actual  answer  to  the  main  question, 
was  formulated  as: 

""The  results  of  the  experiment  are  useful  in  the  determination  of  the  significance  of physical  motion 
in  simulation  of  high  sea  state  for  training  and  doctrine  evaluation”. 

This  proposition  ultimately  needs  to  be  demonstrated  with  evidence  collected  while  preparing  for  the 
experiment  (verification  and  validation  of  the  model  used)  and  running  the  experiment  itself  (verification  and 
validation  of  the  experiment). 

The  outcomes  of  this  case  study  were  publicized  as  a  paper  [44]  and  several  presentations  in  the  SISO 
Simulation  Interoperability  Workshop  and  the  EITSEC  and  ITEC  conference'. 

7,3,1  Real-World  Problem:  Training  Heavy  Weather 

In  order  to  be  able  to  operate  effectively  and  safely,  the  Royal  Dutch  Navy  needs  well  educated  and  trained 
personnel  and  appropriate  doctrine.  Currently  no  specific  training  for  Fleavy  Weather  Ship  Handling 
(HWSH)  is  available.  Learning  to  handle  ships  in  heavy  weather  is  learned  on  the  job  in  real-world  situations 
under  guidance  of  experienced  officers.  The  navy  has  a  lot  of  practical  experience  in  these  real-world 
situations,  but  training  and  doctrine  evaluation  in  a  land-based  simulation  would  be  safer,  cost  effective  and 
time  saving.  Currently  the  navy  uses  a  Full  Mission  Bridge  Simulator  (FMBS)  in  educational  programs, 
but  that  simulator  is  fixed  based.  In  a  fixed-based  simulator  many  of  the  necessary  procedures  can  be 
practiced,  but  how  well  can  one  perform  the  same  procedures  during  real  world  heavy  weather  situations? 

The  Defence  Materiel  Organisation  (DM0)  is  responsible  for  all  materiel  within  the  defence  organization: 
from  procurement  and  major  maintenance  to  disposal.  The  DM0  now  faces  the  question  that  if  the  navy 
wants  to  offer  HWSH  as  part  of  their  educational  program  and  as  a  doctrine  evaluation  aid,  is  the  FMBS 
sufficient?  Part  of  that  question  deals  with  the  issue  of  whether  a  motion  base  is  necessary  for  a  HWSH 
simulator? 


'2011  Euro  SIW,  2011  Fall  SIW,  2011  I/ITSEC,  2012  ITEC. 
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Figure  7-2:  Handling  Ships  in  Heavy  Weather. 


7,3,2  Product  World:  Heavy  Weather  Simulator 

To  answer  the  question  of  DM0,  TNO  designed  an  experiment  to  determine  training  effects  due  to  motion 
simulation  via  an  in-simulator  comparison  approach.  In  this  experimental  design  two  groups  of  test  subjects 
were  used,  one  that  is  trained  with  motion  and  one  without  motion.  All  test  subjects  followed  the  same  test 
sequence  consisting  of  a  habituation  period  followed  by  a  pre-test,  the  training  and  a  post-test.  Both  groups 
did  the  habitation  and  testing  in  the  simulator  with  motion  simulation.  The  scenario  consisted  of  a  number  of 
tasks  that  also  during  heavy  weather  may  still  need  to  be  performed  (following  a  ship,  changing  course, 
making  a  180  turn.)  During  the  experiment  subjective,  Subject-Matter  Expert  (SME),  and  objective 
measurements  were  taken  to  assess  the  test  subject  task  performance. 


Figure  7-3:  The  Full  Motion  Research  Simulator  DESDEMONA. 


Based  on  the  experimental  design  a  Conceptual  Model  (CM)  for  the  M&S  environment  has  been  constructed 
in  cooperation  with  all  parties  together  with  DM0  and  navy  officers  with  relevant  experience.  The  CM 
encompasses  the  ship,  its  environment,  other  traffic  and  tasks  to  be  executed  by  the  ship’s  crew.  The  mine 
hunter  was  chosen  for  this  experiment  because  of  the  size  of  the  ship:  heavy  weather  has  a  large  impact  on 
the  ship’s  motion  and  handling.  A  more  practical  reason  is  that  a  simulation  model  of  the  ship  dynamics  is 
available.  In  Desdemona  only  one  person  can  be  seated.  It  was  chosen  to  use  the  officer  of  the  watch  as  test 
subject,  but  place  them  in  the  position  of  the  helmsman.  The  simulated  sea  state  is  4-6  in  blue  water 
environment. 


7-4 


STO-TR-MSG-073 


CASE  STUDIES 


7,3,3  First  Conclusions 

Based  on  the  case-study  as  executed,  one  can  conclude  that  the  GM-VV  contains  all  necessary  high  level 
ingredients  for  a  rigorous  approach  to  VV&A.  This  abstract  methodology,  however,  needs  to  instantiated  and 
the  instantiated  W&A  method  needs  tailoring  in  order  to  fit  to  the  needs  of  the  W&A  project  at  hand. 
Tailoring  has  been  applied  in  several  ways:  during  instantiation  elements  are  added  or  removed  from  the 
default  GM-VV  and  during  the  execution  of  the  processes  specialization  has  been  applied.  One  of  the  main 
technical  products,  the  V&V  Goal-Claim  Network,  is  built  with  a  continuous  tailoring  by  balancing  results, 
risks,  cost  and  time.  Defining  the  V&V  Experimental  Frame  also  required  extensive  balancing.  The  GM-VV 
tailoring  principles  worked  well  and  resulted  in  a  practical  application  of  the  abstractly  defined  GM-VV. 

Some  parts  of  the  results  of  the  case-study  may  be  re-usable  for  other  VV&A  projects  dealing  with  VV&A 
of  training  simulation  or  experimentation.  Examples  are  the  specialization  of  the  processes  and  parts  of  the 
VV&A  Goal-Claim  Network.  Besides  finishing  the  case-study,  reusable  information  and  lessons-leamed 
from  this  study  have  been  used  to  improve  the  GM-VV  implementation  guide  (Vol.  2). 


7,4  CONCLUSIONS  AND  RECOMMENDATIONS 

Both  case  studies  brought  up  a  number  of  methodological  issues,  textual  issues  related  to  the  documents, 
as  well  as  certain  overall  concerns  about  the  GM-VV  document  set  available  at  time  of  execution.  These 
findings  were  addressed  in  new  releases  of  the  document  set  before  the  SISO  balloting  took/takes  place  in 
2012.  One  issue  that  needs  to  be  addressed  is  the  lack  of  domain  specific  automatic  tool  support  (i.e.  case 
tool).  The  execution  of  a  V&V  process  creates  large  amounts  of  data  that  needs  to  be  collected,  structured 
and  maintained  in  the  course  of  the  process.  The  relations  between  the  data  need  to  be  recorded  and  such  a 
task  needs  automated  tool  support  that  is  currently  missing  in  the  field  (see  Chapter  8).  Using  simple  office 
applications  as  is  often  done  is  more  error  prone  and  simply  not  doable  for  larger  project. 
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Chapter  8  -  NATIONAL  ACHIEVEMENTS  BY  USE  OF  GM-VV 

8.1  EXPERIENCES  FROM  APPLYING  GM-VV  IN  GERMANY 

8.1.1  Preface 

The  following  three  case  studies  summarize  some  experiences  gained  from  application  of  some  essential 
concepts  of  GM-VV  in  different  simulation  projects.  All  three  projects  have  been  initiated  by  institutions  of 
the  German  Armed  Forces  acting  as  project  sponsor  and  as  user.  In  each  case,  companies  specialized  in  the 
respective  field  (szenaris  GmbH,  ESG  GmbH)  had  been  contracted  for  the  design  and  development  of  the 
simulation  models,  and  an  independent  institution  (ITIS)  was  assigned  to  develop  V&V  plans,  to  support 
M&S  documentation,  and  to  support  V&V-activities  in  cooperation  with  sponsor  and  developers.  In  two  of 
these  projects  virtual  simulations  were  implemented  as  training  platforms  while  in  the  third  project  a 
constructive  simulation  environment  had  to  be  developed. 

8.1.2  Case  Study:  Usability  and  Efficiency  of  the  Proposed  Documentation  and  Tailoring 
Concepts  (as  Part  of  the  Project  “Pioneers  Ribbon/Floating  Bridge”) 

In  context  of  the  development  of  a  team  training  simulator  for  coordinating  actions  that  have  to  be  taken  to 
assemble  different  parts  of  a  ribbon/floating  bridge  for  crossing  a  river,  major  experiences  have  been  gained 
from  application  of  a  project  and  model  documentation  process  according  to  national  guidelines  adapted  to 
GM-VV  requirements.  According  to  general  process,  product,  role  and  tailoring  guidelines,  the  project  was 
processed  cooperatively  by  a  team  including  representatives  of  the  project  sponsor  (BWB),  M&S  developers 
(szenaris  GmbH),  and  an  independent  V&V-agent  (ITIS).  While  the  M&S  developers  were  designing  and 
implementing  the  simulation  model  according  to  the  sponsor  needs,  they  had  to  provide  process  and  model 
documentation  for  each  model  development  phase  in  accordance  with  these  guidelines.  In  this  case  study, 
ITIS  provided  guideline  coaching  and  evaluated  the  quality  of  model  documentation  in  form  and  content. 

Activities  which  had  to  be  performed  regarding  GM-VV  and  adapted  national  guidelines: 

•  Coaching  the  application  of  the  process  and  documentation  guideline; 

•  Supporting  the  tailoring  of  M&S  process  and  documentation  activities; 

•  Sample  application  of  V&V  techniques  for  quality  control  and  for  effort  or  cost  estimation  of  V&V- 
activities  and  documentation;  and 

•  Gathering  feedback  from  sponsor  and  developers  regarding  the  general  documentation  structures 
and  guideline  concept. 

Major  findings  and  lessons  learned: 

•  Basically,  the  proposed  documentation  structures  and  guideline  was  perceived  as  time-consuming 
but  also  as  beneficial  by  the  M&S  developer.  After  this  cause  study,  the  company  decided  to  apply 
this  guideline  in  other  projects  on  its  own  initiative. 

•  Standardized  formalization  languages,  methods,  and  tools  are  indispensable  for  model 
documentation. 

•  IPR  and  know-how  protection  has  been  seen  as  a  critical  issue  related  to  model  documentation  and 
(independent)  model  V&V. 

•  Costs  and  efforts  for  model  documentation  and  model  V&V  should  be  calculated  as  early  as 
possible  along  with  tailoring  decisions.  As  total  project  budget,  as  well  as  quality  assurance, 
will  significantly  depend  on  tailoring  decisions,  these  should  be  accepted  and  well  documented  by 
all  contributing  parties. 
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Figure  8-1:  Pioneers  Ribbon/Fioating  Bridge. 


8,1,3  Case  Study:  Efforts,  Efficiency  and  Effectiveness  of  Performing  V«&V- Activities 
(as  Part  of  the  Project  “Robot  Movement  Simulator”) 

While  the  first  case  study  was  focused  on  the  aspects  like  usability,  efforts  for  and  efficiency  of  the  proposed 
documentation  guideline  and  tailoring  concept,  the  goal  of  this  case  study  was  to  investigate  acceptance, 
feasibility  and  involved  efforts/costs  for  development  and  application  of  a  V&V-plan  in  the  context  of  the 
robot  simulator  project.  For  budgetary  reason,  the  scope  of  this  case  study  was  limited,  or  tailored  to  a  sub¬ 
set  of  the  model  development  phases,  and  to  three  of  thirteen  pre-defmed  application  scenarios. 

Major  activities  performed  in  context  of  this  case  study  were: 

•  Supporting  the  required  model  documentation  (e.g.  regarding  completeness,  consistency); 

•  Development  of  a  V&V-plan  as  a  result  of  static  tailoring  based  on  the  project  requirements  as  well 
as  based  on  case  study  budget  and  in  cooperation  with  sponsor,  developer  and  V&V-agent; 

•  Cooperative  decisions  regarding  additional  dynamic  tailoring  actions  (together  with  the  sponsor  and 
the  developer); 

•  Execution  of  V&V  activities  and  documentation  of  V&V  results  according  to  the  V&V  plan;  and 

•  Sample  application  of  different  V&V  techniques  to  investigate  their  effectiveness. 

Major  findings  and  lessons  learned  are: 

•  Time  and  efforts  required  for  the  initial  application  of  the  documentation  guideline  is  extremely  high 
(especially  for  the  M&S  developer  who  rarely  documents  their  work).  However,  once  such  a 
guideline  is  properly  introduced  to  the  M&S  development  team,  documentation  time/costs  for 
further  projects  can  be  significantly  reduced. 
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•  A  documentation  guideline  and  structures  in  the  form  of  flexible,  “living”  guidelines  and  documents 
would  increase  efficiency.  Consequently,  two  studies  for  development  of  electronic  guidelines  were 
initiated  and  conducted. 

•  The  “V&V  Triangle”  concept  can  be  applied  for  process  management  tasks  (such  as  planning, 
organizing,  and  monitoring  the  V&V  effort)  as  well  as  for  technical  tasks  (like  analyzing,  evaluating, 
reviewing,  and  testing  model  elements). 

•  Both  static  tailoring  (at  the  beginning  of  an  M&S  project)  and  dynamic  tailoring  (adaptation  during 
the  M&S  development  process)  are  indispensable. 

•  Regarding  IPR  and  know-how  protection,  the  following  concept  was  proposed: 

1)  The  independent  V&V  agent  specifies  detailed  V&V  requirements,  examination  criteria  and 
measures  and  contents  of  V&V  reports  for  model  work  products  subject  of  V&V; 

2)  An  internal  inspector  on  the  M&S  developer  side  (e.g.  from  the  quality  assurance  department) 
performs  the  specified  examination  according  to  the  V&V  requirements,  and  documents  the 
V&V  process  and  its  results;  and 

3)  The  V&V  agent  evaluates  correctness  and  validity  of  the  model,  simulation  and  data  based  on 
the  created  examination  protocol  of  the  internal  inspector. 


Figure  8-2:  Robot  Movement  Simulator. 


8,1,4  Case  Study:  V&V  of  Safety-Critical  Constructive  Simulation  Model 
(as  Part  of  the  Project  “Guidelines  for  Platoon  Commanders”) 

Goal  of  this  simulation  project  was  to  be  able  to  analyze  different  guideline  options  for  platoon  Commanders, 
how  to  lead  his  or  her  platoon  in  different  and  safety-critical  scenarios.  One  of  the  project  requirements 
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defined  a  limited  time  frame  for  simulation  model  development  and  availability  of  the  requested  simulation 
results.  Beside  this  time  limitation,  quality  of  the  modeling  process  and  its  results  should  be  evaluated 
through  V&V.  Therefore,  as  a  case  study,  ITIS  as  an  independent  V&V  agent  was  assigned  to  guide  or 
support  some  V&V  activities  concurrently  with  simulation  model  development. 


Figure  8-3:  Platoon  Commanders. 

Major  activities  performed  by  the  V&V  agent  included: 

•  Supporting  the  preparation  of  the  model  documents  especially  checking  completeness  and 
consistency); 

•  Tailoring  of  M&S  activities  and  documentation  (in  cooperation  with  the  project  sponsor 
and  the  M&S  developer); 

•  Preparation  of  the  V&V  plan  (in  cooperation  with  the  project  sponsor  and  the  M&S  developer); 

•  Selecting  V&V  techniques  and  conducting  V&V  activities  according  to  the  V&V  plan;  and 

•  Documentation  and  demonstration  of  the  V&V  results. 

Major  findings  and  lessons  learned  have  been: 

•  The  generic  tailoring  concept  of  GM-VV  was  applied  and  has  confirmed  its  usability  as  a  general 
framework.  In  total,  applying  static  and  dynamic  tailoring  actions  according  to  the  project 
requirements  and  constraints,  8  versions  of  the  V&V  plan  were  developed.  The  first  version  was 
prepared  based  on  static  tailoring,  all  the  others  were  results  of  dynamic  tailoring  needs. 
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•  Because  of  some  vague  Sponsor  Need  specifications  and  a  large  number  of  uncertainties  (e.g.  as  an 
extensive  involvement  of  Subject-Matter  Experts  (SMEs)  was  indispensable),  the  focus  of  V&V 
was  directed  towards  an  evaluation  of  the  Structured  Problem  Description  (SPD)  and  the  Conceptual 
Model  (CM)  development.  As  a  consequence  of  V&V  of  work  products  and  documentation  of  these 
two  development  phases,  and  based  on  intensive  discussions  between  the  V&V  agent,  developer  and 
sponsor,  9  versions  of  SPD  and  10  versions  of  CM  had  to  be  developed. 

•  A  combination  of  different  V&V  techniques  (e.g.  combination  of  Inspections,  Face  Validation, 
and  Visualization/Animation  in  this  case  study)  was  very  useful  and  demonstrated  to  be  very 
effective. 

Time  and  efforts  required  for  execution  of  V&V  techniques  and  interpretation  of  V&V  results  has  to  be 
calculated  at  the  begin  of  a  project  (static  tailoring  based  on  project  constraints  and  requirements). 
Experience  showed  that  just  time  scheduling  requirements  and  availabilities  of  team  members  (roles),  has  to 
be  considered  as  a  limiting  factor  for  processing  certain  V&V  activities.  This  experience  indicated  the  need 
for  dynamic  tailoring  actions. 


8.2  EXPERIENCES  FROM  APPLYING  GM-VV  IN  SWEDEN 
8,2,1  Case  Study:  Air-to-Air  Missile 

This  case  study  is  based  on  [37]  reporting  on  a  case  where  the  GM-VV  has  been  tested  within  the  Swedish 
Defence  Research  Agency  (FOI)  on  an  unclassified  model  of  a  generic  air-to-air  missile.  The  intended  use  of 
this  missile  model  was  training  of  fighter  pilots  in  Beyond-Visual-Range  (BVR)  scenarios,  thus  aimed  at 
being  Suitable  for  Air  Combat  Training. 

The  exercise  involved  a  team  of  M&S  developers  using  requirements  specifications  on  this  generic  model 
coming  from  project  sponsor  organizations  such  as  the  Swedish  Armed  Forces  and  industrial  partners. 
The  GM-VV  has  been  used  to  refrne  and  elaborate  on  these  requirements  specifications  to  derive 
an  acceptance  recommendation  for  the  intended  use  of  the  model.  Three  top  requirements  on  the  model  have 
been  selected  to  test  and  apply  the  GM-VV  (Figure  8-4): 

•  Sufficient  Realism:  Choices  of  physical  modelling  in  terms  of  structure  and  sub  models  reflecting 
the  vital  components  of  a  BVR  missile. 

•  Sufficient  Transparency:  Information  regarding  the  interaction  between  the  missile  and  all 
surrounding  objects  (target,  environment,  etc.)  to  be  displayed  to  the  pilot  during  flight  but  also  for 
post  mission  review. 

•  Be  Executable  in  a  Simulator:  Use  of  MERLIN,  a  component  based  simulation  framework  for  soft 
real-time  simulations  of  weapon  platforms,  to  run  the  missile  model. 
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Figure  8-4:  Requirements  Breakdown  for  the  Air-to-Air  Missiie  Modei. 


Objectives  of  the  exercise  regarding  the  GM-VV: 

•  Test  the  applicability  of  the  methodology; 

•  Coach  the  application  of  the  GM-VV  process; 

•  Support  the  development  of  the  Goal-Claim  Networks; 

•  Support  the  tailoring  of  M&S  process  and  documentation  activities;  and 

•  Gather  feedback  from  M&S  developers  regarding  utility  and  benefits  of  using  the  GM-VV, 
highlighting  particular  aspects  of  the  methodology  being  more  relevant/useful  as  well  as  ones  that 
are  less  relevant/useful  that  need  be  improved. 

Major  findings  and  lessons  learned: 

•  GM-VV  is  a  promising  method  for  the  validation  of  missile  models,  and  should  also  be  suitable  for 
other  applications.  A  strength  of  the  method  is  the  clear  link  to  the  application  and  targets  with  the 
model. 

•  With  a  structured  V&V  methodology  created,  a  documented  and  transparent  chain  of  evidence 
between  the  declared  objective  of  the  model  and  the  assertion  that  the  model  lives  up  to  the  goal. 

•  The  importance  of  a  thorough  requirements  definition  and  formulation  of  the  intended  use  of  the 
model  has  been  clarified  in  the  work.  In  the  reported  example  it  has  been  found  that  a  simpler  model 
of  the  missile,  could  have  had  fulfilled  the  use  area  so  as  described  herein. 

•  The  work  carried  out  has  also  further  demonstrated  the  utility  of  a  close  dialogue  between  V&V 
implementers,  M&S  developers  and  users. 


8.3  EXPERIENCES  FROM  APPLYING  GM-VV  IN  THE  NETHERLANDS 

8,3,1  Case  Study:  Distributed  Air  Operation  Mission  Training  Environment 

This  example  is  based  on  [38]  and  [39].  In  order  to  be  able  to  operate  effectively  and  safely,  the  Royal 
Netherlands  Air-Force  (RNLAF)  Air  Operation  Control  Station  (AOCS)  needs  fighter  controllers  that  are 
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familiarized  with  F-16  cockpit  operations.  Until  recently,  fighter  controllers  were  trained  for  this  purpose  by 
several  familiarization  flights  in  the  RNAF  F-16B  dual-seat  aircraft.  For  economic  reasons,  the  RNLAF  has 
phased  out  all  its  F-16B  aircraft.  Therefore,  AOCS  has  to  find  alternatives  to  continue  this  type  of  fighter 
controller  training.  A  simulation-based  training  system  could  provide  the  fighter  controller  a  cost-effective 
and  safe  learning  environment  in  which  fighter  controllers  can  experience  F-16  flight  operations  in  realistic 
mission  scenarios.  To  support  the  concept  development  and  specification  phase  of  such  future  simulation 
training  environment,  a  prototype  was  developed  by  the  Dutch  National  Aerospace  Laboratory  NLR. 
This  prototype  integrates  NLR’s  four  existing  fixed-based  F-16  flight  simulators,  a  constructive  simulation 
that  provides  computer  generated  air  threats,  and  two  real-life  and  NATO  classified  fighter  controller 
working  stations  (MASE)  into  a  single  distributed  air  operation  mission  training  environment.  Interoperability 
between  all  simulation  assets  was  accomplished  through  DIS. 

A  V&V  study  was  requested  to  assess  the  utility  and  validity  of  this  training  simulation  concept  and  its 
underlying  technology  for  intended  use  of  AOCS.  Based  on  the  acceptance  recommendations  of  this  V&V 
study  AOCS  would  take  decisions  regarding  the  acquisition  process  of  a  new  alternative  training 
environment  (e.g.  go/no-go  decision  for  an  M&S  system  or  M&S  requirements  refinements).  The  GM-VV  is 
the  recommended  standard  for  V&V  within  the  Dutch  MoD.  Therefore  this  methodology  was  selected  for 
this  study.  Since  AOCS  didn’t  have  the  GM-VV  knowledge  and  expertise  they  contracted  (i.e.  through  the 
GM-VV  recommended  V&V  agreement)  the  Dutch  V&V  service  supplier  organization  Q-tility  to  perform 
the  V&V.  As  a  V&V  User/Sponsor  AOCS  was  actively  involved  in  the  V&V  work  (e.g.  review  of  the  V&V 
argumentation  structure,  V&V  plan  approval  and  SME  provision  in  the  V&V  execution  phase)  and  was  kept 
up  to  date  on  the  V&V  progress  by  Q-tility  though  regular  V&V  progress  reports. 


Figure  8-5:  Distributed  Air  Operation  Mission  Training  Environment  Architecture. 


Objectives  of  the  exercise  regarding  the  GM-VV: 

•  Evaluate  the  utility  of  the  GM-VV  project  and  technical  level  components  for  the  V&V  of  (EVC) 
distributed  simulation  applications; 

•  Establish  a  tailored  instance  of  GM-VV  in  the  form  of  a  reusable  V&V  life-cycle  model  by  applying 
the  GM-VV  tailoring  framework  Phase  1:  instantiation  [2]; 

•  Evaluate  and  refine  the  yEd  tool  for  V&V  argumentation  structure  and  standard  office  tools 
(e.g.  MS-Word,  MS-Excel,  MS-Project  and  MS-Sharepoint)  to  support  the  GM-VV  process 
activities,  document  and  manage  the  GM-VV  information  artifacts;  and 
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•  Develop  an  initial  basis  for  a  M&S  risk-analysis  and  reporting  approach  that  could  be  applied  as 
future  concrete  technique  for  “tailoring  by  balancing”  in  the  GM-VV  tailoring  framework  Phase  2: 
optimization  [2]. 

Major  findings  and  lessons  learned: 

•  The  GM-VV  project  and  technical  level  components  showed  sufficient  utility  for  a  satisfactory 
V&V  of  this  (LVC)  distributed  simulation  application. 

•  The  yEd  tool  and  the  standard  office  application  based  GM-VV  support  tools  showed  to  be  effective 
and  efficient  for  this  V&V  study. 

•  The  V&V  of  the  classified  simulation  application  MASE  required  additional  V&V  planning,  time 
and  resources  to  complete  the  V&V  study  compared  to  the  non-classified  simulation  applications. 
Furthermore,  V&V  configuration  and  information  management  process  required  more  attention, 
effort  and  more  formal  implementation. 

8,3,2  Case  Study:  Public  Order  Management  Serious  Game 

This  example  is  based  on  [40].  TNO  examined  how  Commanders  can  learn  to  maintain  public  order  in  their 
area  of  operations.  Serious  gaming  proves  to  be  an  effective  and  efficient  learning  tool.  Q-tility  has 
performed  an  explorative  V&V  study  to  the  added  value  of  a  demonstrator  Public  Order  Management 
(POM)  serious  game.  The  study  results  show  its  added  value  as  well  as  extensions  to  the  current  game. 

One  of  the  POM  game  users  is  the  Royal  Netherlands  Marechaussee  (KMar).  The  game  can  potentially  be 
used  for  a  number  of  training  objectives.  With  V&V  the  KMar  instructors  want  to  find  for  what  objectives 
the  game  is  already  suited,  and  what  needs  to  be  adapted  in  order  for  the  game  to  have  utility  for  the  full  set 
of  training  objectives. 

The  added  value  of  Q-tility  V&V  approach  is  that  it  takes  more  than  just  the  technical  game  aspects  into 
account:  they  also  examine  and  give  advice  on  the  required  personnel,  effective  and  efficient  use  of  a  game 
in  combination  with  other  available  training  methods  and  even  gaming  relevant  policies  that  can  enhance 
the  added  value  of  the  game. 


Figure  8-6:  Public  Order  Management  Serious  Game. 
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Objectives  of  the  exercise  regarding  the  GM-VV: 

•  The  KMar  trainers  and  various  subject  matter  experts  hired  by  Q-tility  derived  the  acceptance 
criteria  from  the  main  goal:  the  game  must  ensure  that  training  objectives  are  achieved  more 
efficiently  and  effectively; 

•  Further  define  templates  for  GM-VV  documents; 

•  The  V&V  tests  consisted  of  two  full  training  sessions,  4  days  in  total,  in  which  KMar  personnel 
used  the  POM  game  as  they  intend  to  use;  and 

•  V&V  techniques  included  interviews,  observations,  360°  assessment,  and  hardware/software 
inspections. 

Major  findings  and  lessons  learned: 

•  The  V&V  results  confirmed  many  of  the  strengths  and  weaknesses  of  the  game. 

•  The  added  value  of  V&V  is  that  now  sufficient  -  and  independently  obtained  -  data  is  available  to 
back  up  these  claims. 

•  The  POM  game  developers  also  appreciated  the  independent  view  on  the  usefulness  of  the  game  to 
prevent  tunnel  vision  in  the  development  team. 

•  Immediately  after  the  V&V  tests  the  KMar  has  started  implementing  changes  to  allow  for  efficient 
and  effective  use  of  the  POM  game. 
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9.1  OVERVIEW,  CURRENT  SITUATION  AND  RECOMMENDATIONS 

The  case  studies  represented  in  the  previous  paragraph  identified  a  need  for  dedicated  tools  that  will  support 
performing  VV&A  activities.  Several  issues  need  to  be  supported;  predominantly  the  large  amount  of  data 
being  collected  and  maintained  during  a  V&V  activity  is  driving  this  requirement.  The  other  issue  to  cover  is 
to  be  able  to  enforce  a  well-structured  and  formalized  approach  in  data  collection.  The  subsequent  need  to 
provide  an  inifastructure  to  document,  maintain,  re-asses  and  support  version  control  the  evidence  as  well  as 
the  process  of  building  the  V&V  argumentation  was  recognized  at  a  very  early  stage  of  the  project. 
First  efforts  in  this  field  were  made  within  REVVAl  in  2002,  which  generated  a  list  of  available  tools  that 
might  support  the  V&V  process.  The  conclusion  after  completion  of  this  effort  was  that  hardly  any  tools 
existed,  a  situation  that  unfortunately  hasn’t  much  improved  since. 

The  US  DoD  MS&CO,  the  successor  of  US  DoD  DMSO,  recently  developed  a  standard  document  template 
form  (Military  Standard  3022  [30]).  These  templates  prescribe  a  common  documentation  format. 
This  standard  contains  a  set  of  four  documents  that  subsequently  cover  the  Accreditation  Plan,  the  V&V 
Plan,  the  V&V  Report,  and  the  Accreditation  Report.  The  US  Navy  M&S  Office  (NMSO)  developed  a  DoD 
VV&A  Documentation  Tool  (DVDT)  to  automate  the  set  of  MIL-STD-3022  templates. 

MSG-073  product  GM-VV  is  based  on  readily  available  standard  engineering  approaches  however  differs  in 
terms  of  collecting  evidence  and  providing  proof  of  a  sound  simulation  solution  with  regard  to  the  ‘intended 
use’.  The  requirement  for  automated  V&V  support  is  driven  by  V&V  specific  needs  which  led  to 
investigating  approaches  within  the  security  and  safety  world.  These  communities  have  to  deal  with  the  same 
problem  as  the  V&V  community  in  dealing  with  risk.  The  Goal  Structured  Notation  (GSN)  and  Claim 
Argument  Evidence  (CAE)  structures  utilized  in  these  communities  seems  to  have  a  close  fit  the  VV&A 
needs.  One  company,  Adelard,  developed  the  Assurance  and  Safety  Case  Environment  (ASCEtm),  in  which 
the  needed  formalism  is,  however,  lacking.  Similar  approaches  are  being  under  development  within  the 
academic  world,  however  with  little  success  so  far.  In  daily  practice  standard  MS  Office  products  are 
generally  used  for  documentation  of  V&V  activities.  Requirements  management  tools  like  DOORS  or 
RequisitePro  may  have  been  used  as  an  alternative. 

VV&A  for  simulations  recently  has  become  more  prominently  visible.  To  make  a  real  breakthrough, 
however,  the  V&V  process  needs  more  efficient  tools  to  proof  its  real  value.  Being  supported  by  better  and 
more  automated  tools  would  surely  help  to  make  VV&A  not  only  more  efficient  and  effective  but  also  lower 
the  financial  threshold  for  investing  in  V&V  activities.  NATO  should  therefore  invest  in  the  promotion  of 
VV&A  and  challenge  industry  to  invest  in  V&V  support  by  respective  CASE-tools. 
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Chapter  10  -  CONCLUSIONS  AND  RECOMMENDATIONS 

The  MSG-073  has  successfully  completed  its  TOR  as  it  was  approved  in  2009.  The  GM-VV  series  of 
standards  is  the  first  standardization  effort  completed  in  SISO  with  the  support  of  the  NMSG. 

This  final  report  is  only  shortly  describing  the  methodology:  the  description  provided  is  better  described  in 
the  3  SISO  documents.  Information  reproduced  from  these  documents  has  SISO  copyrights:  SISO  kindly 
granted  permission  to  reproduce  a  part  of  them  for  this  report.  Obviously  all  NATO  members  and  partners 
are  freely  allowed  to  download  the  3  documents  from  the  SISO  website  (http://www.sisostds.org). 
This  standardization  effort  was  the  main  part  of  the  activities  of  MSG-073  and  it  was  hopefully  completed  in 
November  2013.  The  GM-VV  description  was  already  included  in  the  current  version  of  the  NATO  M&S 
Standards  Profile  [31]  and  has  been  recently  updated  for  the  publication  of  a  new  version  (Version  C). 

One  objective  of  the  MSG-073  was  the  dissemination  of  the  GM-VV  concepts  and  methodology:  it  has  been 
achieved  by  numerous  papers,  talks  and  tutorials  that  were  given  by  MSG-073  members  during  diverse 
conferences  (including  NATO  Symposia),  SISO  workshops  and  important  M&S  events  like  the  ITEC  and 
IITSEC.  References  on  recent  GM-VV  publications  may  be  found  in  the  list  of  references  of  this  final  report. 
This  tutoring  activity  will  be  complemented  by  a  Lecture  Series  that  will  take  place  in  2014  (MSG-123). 
Members  of  the  MSG-073  and  contributing  nations  realize  that  this  effort  has  to  be  maintained  and  they  are 
ready  to  continue  the  effort  under  adequate  formats  such  as  workshops,  specialist  team  meetings,  etc. 

As  the  V&V  activity  is  complex  and  painful,  there  are  still  a  lot  of  aspects  to  be  explored  or  improved. 
Then  the  MSG-073  Nations  will  propose  a  new  NMSG  activity  on  the  risk  identification  and  management  in 
using  M&S  systems.  NATO  Nations  and  Partners  are  invited  to  participate  in  this  new  NMSG  activity. 

The  GM-VV  may  appear  as  very  theoretical  but  the  methodology  was  developed  and  matured  using 
experiences.  Some  use  cases  are  reported  in  this  report.  More  details  could  be  found  in  references  of 
published  documents.  One  of  the  outputs  of  past  experiences  was  the  need  for  developing  specific  tools  to 
support  the  use  of  GM-VV:  people  are  using  existing  tools,  which  are  more  or  less  fitted  to  the  activity, 
but  there  is  a  real  need  to  develop  better  adapted  tools.  Despite  this  was  not  listed  as  an  objective  of 
MSG-073,  this  question  was  treated  by  the  group  without  any  success.  Establishing  specification  for 
GM-VV  tooling  does  not  appear  difficult  but  it  was  not  done  for  a  lack  of  resources  during  the  MSG-073 
time  life. 

Nations  are  invited  to  report  on  future  uses  they  will  have  on  GM-VV  taking  profit  of  future  NMSG 
activities.  Experience  will  allow  the  methodology  to  evolve  and  improve.  Concerning  the  standardization 
aspect,  standards  are  not  built  one  time  and  forever.  Then  there  is  a  need  to  monitor  uses  of  the  methodology, 
lessons  learned  from  future  uses  to  identify  needed  update  to  the  first  version  of  the  GM-VV.  Support  for  the 
set  of  GM-W  documents  will  be  done  via  SISO.  A  SISO  Product  Support  Group  (PSG)  has  been  established 
in  2013  for  the  GM-VV.  It  is  recommended  that  nations  mandate  representatives  to  the  SISO  PSG  to  ensure 
that  the  methodology  stay  up-to-date  and  evolve  according  to  the  state-of-the-art. 
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TECHNICAL  ACTIVITY  PROPOSAL  (TAP) 


ACTIVITY 

Task  Group 

Generic  Methodology  for  Verification  and 
Validation  of  Models,  Simulations  and  Data 
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2012 
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KEYWORDS 

M&S 

Data 

VV&A 

Interoperability 

Re-use 

BACKGROUND  AND  JUSTIFICATION  (RELEVANCE  TO  NATO) 

The  general  issue  of  Verification  and  Validation  (V&V)  and  Acceptance  (VV&A)  of  Models,  Simulations, 
and  Data  has  been  addressed  in  many  working  groups  inside  and  outside  NATO.  While  there  is  a  general 
consensus  on  the  importance  of  this  topic,  there  are  few  international  standards  on  VV&A.  With  the  start  of 
important  NATO  training  projects  like  the  ACT  Snow  Leopard,  Nations  will  have  the  opportunity  of 
contributing  to  the  supporting  simulation  systems.  The  application  of  M&S,  such  as  within  the  training 
domain,  brings  an  inherent  risk  associated  with  the  danger  of  using  erroneous  or  unsuitable  models  and 
simulation  results.  Verification  and  Validation  (V&V)  of  models  and  simulations  are  intended  to  ensure  that 
only  correct  and  suitable  results  are  used  thereby  facilitating  risk  management.  There  is  a  need  to  agree  on  a 
generic  methodology  to  validate  models,  simulations  and  data  to  be  used  in  future  NATO  and  national 
projects.  A  neutral  and  consensus-agreed  VV&A  guidance  document  shall  be  proposed  to  NATO. 

OBJECTIVE(S) 

•  To  finalize  the  work  on  a  VV&A  document  set  under  development  by  the  SISO  GM-VV  Product 
Development  Group. 

•  To  guide  the  proposed  document  set  through  the  formal  SISO  standardization  process. 

•  To  participate  in  the  review,  comment  and  balloting  efforts  for  these  new  SISO  standards. 

•  To  assemble  confidence  in  and  fine-tune  the  methodology  on  the  basis  of  ‘real’  case  studies. 

•  To  provide  education  and  training  based  on  a  documentation  set  for  dissemination. 
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TOPICS  TO  BE  COVERED 

Main  outputs  of  the  TG  will  be  finalized  under  3  SISO  “guidance”  documents:  the  “Handbook”:  a  user’s 
manual,  the  “Reference  manual”;  which  will  document  the  underlying  concepts  of  the  methodology,  and  the 
“Recommended  Practice  Guide”  which  provides  specific  guidance  with  regards  to  the  selection  and  use  of 
techniques. 


DELIVERABLE  AND/OR  END  PRODUCT 

The  final  technical  report  published  under  a  RTO  reference  will  be  a  short  introduction  to  the  3  SISO 
guidance  documents  and  recommendations  for  their  specific  use  in  NATO. 


TECHNICAL  TEAM  LEADER  AND  LEAD  NATION 

France  will  chair  the  Task  Group. 


NATIONS  WILLING  TO  PARTICIPATE 

Participating  Nations  include  Canada,  Denmark,  France,  Netherlands,  and  Sweden.  Other  NATO  and  Partner 
Nations  are  kindly  invited  to  join  the  Task  Group. 


NATIONAL  AND/OR  NATO  RESOURCES  NEEDED 

Input  to  and  participation  in  the  meetings  including  travel  funds  (1-2  specialists  per  Nation/NATO  body). 


RTA  RESOURCES  NEEDED  (E.G.  CONSULTANT  FUNDING) 

Electronic  publication.  One  practical  case  for  experimentation  of  the  methodology  could  be  proposed  the  last 
year  of  this  activity. 
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TERMS  OF  REFERENCE  (TOR) 


ORIGIN 

Background 

The  general  issue  of  Verification  and  Validation  (V&V)  and  Acceptance'  (VV&A)  of  Models,  Simulations, 
and  Data  has  been  addressed  in  many  working  groups.  While  there  is  a  general  consensus  on  the  importance 
of  this  topic,  there  are  few  international  standards  on  VV&A  and  they  are  targeted  on  simulation 
interoperability  within  distributed  systems  of  simulation  (federations),  and  related  to  both  DIS  and  HLA 
standards  (IEEE  1278.4  and  1516.4).  NATO  has  been  supportive  to  the  IEEE  HLA  VV&A  efforts  under  two 
Task  Groups  (MSG-019  and  MSG-054). 

VV&A  of  Models  and  Simulation  (M&S)  composing  distributed  systems  are  supported  by  national 
processes,  methodology  and  tools  (like  the  US  VV&A  Recommended  Practice  Guidance,  RPG),  but  there  is 
clearly  a  lack  of  internationally  recognized  standards  on  this  general  topic.  As  early  as  2002,  an  European 
consortium  was  created  to  address  this  issue  under  the  Western  European  Armament  Group  (WEAG) 
umbrella:  the  REWA  (Reference  project  for  W&A)  project  was  bom.  In  its  final  composition, 
this  consortium  was  composed  from  5  NATO  Member/Partner  Nations  (Canada,  Denmark,  France, 
Netherlands  and  Sweden)  and  has  started  a  standardization  effort  in  the  context  of  SISO  (the  Simulation 
Interoperability  Standards  Organization).  Unfortunately  the  REVVA  contract  ends  in  2008  and  the 
consortium  will  have  to  be  disbanded  before  the  standardization  effort  is  finalized  as  WEAG  is  no  longer 
existing  as  a  parent  organization. 

Since  all  Nations  contributing  to  the  REVVA  efforts  are  represented  in  the  NATO  M&S  Group  (NMSG)  it 
seemed  natural  that  the  NMSG  take  the  lead  on  supervising  the  corresponding  V&V  effort  currently  under 
process  in  SISO.  This  will  be  the  opportunity  to  welcome  additional  nations  to  support  this  effort.  This  is 
also  clearly  in  consistency  with  the  NMSG  standardization  mission  and  its  cooperation  with  SISO. 

Justification  (Relevance  to  NATO) 

Credibility  and  validity  are  critical  to  the  effective  and  appropriate  use  of  models  and  simulations  (M&S) 
whether  used  independently  or  combined  in  federations.  First  W&A  efforts  were  the  responsibility  of 
former  TGs  of  the  NMSG  with  a  well-targeted  focus  on  federation  of  simulations.  With  the  start  of  important 
NATO  training  projects  like  Snow  Leopard,  Nations  have  the  opportunity  of  contributing  to  the  supporting 
simulation  systems.  There  is  a  need  to  agree  on  a  generic  methodology  to  validate  models,  simulations  and 
data  to  be  used  in  future  NATO  and  national  projects.  A  neutral  and  consensus-agreed  guidance  document 
shall  be  proposed  to  NATO. 

Military  Benefits 

Increasingly,  M&S  is  being  exploited  as  an  enabling  technology  to  support  tactical,  operational  and  strategic 
objectives  within  the  military  domain.  The  use  of  M&S  provides  those  within  the  military  domain  with  a 
powerful  and  resource-efficient  capability  for: 

•  Training  and  education; 

•  Mission  analysis  and  rehearsal; 

•  Decision  support  (such  as  exploration  of  doctrinal  alternatives); 


Acceptance  is  the  decision  to  use  a  simulation  for  a  specific  purpose  and  the  term  Accreditation  is  the  official  certification 
that  a  model  or  simulation  is  acceptable  for  use  for  a  specific  purpose. 
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•  Capability  management; 

•  Investigation  of  leading-edge  technologies;  and 

•  Effective  support  to  the  acquisition  process. 

The  application  of  M&S,  such  as  within  the  training  domain,  brings  an  inherent  risk  associated  with  the 
danger  of  using  erroneous  or  unsuitable  models  and  simulation  results.  Verification  and  Validation  (V&V)  of 
models  and  simulations  are  intended  to  ensure  that  only  correct  and  suitable  results  are  used  thereby 
facilitating  risk  management. 

Lots  of  military  decisions  could  be  based  on  M&S  results.  One  of  the  aims  of  the  V&V  methodology  is  to 
structure  the  argumentation  which  will  permit  the  military  authorities  to  take  decisions  based  on  a  more 
credible  evaluation  of  the  risks  involved. 

The  non-direct  benefits  concern  every  phase  of  the  design  and  development  processes  of  military  systems 
and  weapons  by  armament  companies  and  their  customers. 


OBJECTIVES 

Area  of  Research  and  Scope 

The  proposed  Task  Group  will  build  on  former  technical  activities: 

•  Former  NMSG  TGs  (MSG-0 1 9  and  MSG-054); 

•  Previous  work  on  VV&A  under  the  International  Test  Operations  Procedures  (ITOP)  organization; 

•  Past  work  on  the  previous  GM-VV  SISO  Study  Group  on  generic  Methodology  for  V&V 
(GM-VV);  and 

•  Current  activity  of  the  current  GM-VV  SISO  Product  Development  Group  (PDG). 

The  TG  will  act  with  the  GM-VV  PDG  and  will  be  in  direct  support  to  the  Drafting  Group  of  the  PDG. 
More  specifically  the  TG  will  be  in  charge  of  supervising  the  PDG  activities  to  guarantee  that  NATO  and 
Member/Partner  Nation’s  needs  are  addressed  in  the  PDG.  Moreover  SISO  PDGs  are  widely  open  to  other 
communities  of  interest  such  as  transport,  medicine,  etc.,  and  not  only  targeted  to  military  or  security 
purposes. 

Specific  Activities  to  be  Performed  by  the  TG 

The  major  objective  of  this  TG  is  to  create  a  follow-on  process  for  finalizing  the  standardization  work  on 
generic  VV&A  for  models  and  simulations.  The  TG  will  be  responsible  and  engaged  in  specific  activities 
with  the  following  objectives: 

•  To  finalize  the  work  on  the  VV&A  document  set  (this  task  started  under  the  European  REWA 
consortium  banner  and  is  now  under  development  by  the  SISO  GM-VV  PDG); 

•  To  guide  the  proposed  document  set  through  the  formal  SISO  standardization  process; 

•  To  participate  in  the  review,  commenting  and  balloting  efforts  needed  for  creation  of  a  new  SISO 
standard; 

•  To  assemble  confidence  in  and  fine-tune  the  methodology  on  the  basis  of  ‘real’  case  studies;  and 

•  To  provide  education  and  training  based  on  a  documentation  set  for  dissemination. 
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Other  objectives  to  be  fulfilled  cover  the  inclusion  of  the  SISO  balloted  products  within  the  NATO  M&S 
Standards  Profile  (NMSSP)  and  to  produce  a  final  Technical  Report.  Through  case  studies  the  methodology 
will  be  refined  in  the  course  of  this  TG,  while  at  the  same  time  providing  material  for  dissemination  and 
training. 

Products 

Main  outputs  of  the  TG  will  be  published  under  SISO  references  according  to  the  Technical  Cooperation 
Agreement  between  NATO  and  SISO: 

•  The  “Handbook”,  which  will  be  a  User’s  Manual  that  safely  guides  its  users  through  the  VV&A 
effort  and  clarifies  their  responsibilities  by  explaining  how  to  apply  the  methodology  in  practice;  it 
describes  the  activities  to  perform  and  the  products  to  produce,  the  interactions  taking  place  among 
those  involved,  the  flow  of  products,  and  how  to  tailor  the  methodology  to  the  specific  needs  of  the 
M&S  project. 

•  The  “Reference  Manual”,  which  will  document  the  underlying  concepts  of  the  methodology, 
including  the  foundations  of  the  chosen  terminology,  the  explanation  of  the  dependencies  between 
activities  and  products,  their  meaning  for  the  VV&A  endeavor,  and  the  rationale  for  their  execution 
and  creation;  the  Reference  Manual  is  referred  to  whenever  a  deeper  understanding  of  the 
methodology  is  required.  The  Reference  Manual  will  provide  all  the  handles  needed  for  future 
development  of  supportive  case-tools  and  repositories. 

•  The  “Recommended  Practice  Guide”,  which  will  be  a  recommended  practices  document  that 
provides  specific  guidance  with  regards  to  the  selection  and  use  of  techniques  and  tools  in  support  of 
the  user  manual  (the  Handbook);  it  will  include  domain  specific  case  studies  thereby  illustrating  the 
application  and  tailoring  of  the  methodology.) 

The  first  and  the  third  document  will  be  “officially  balloted”  SISO  “guidance  products”;  the  second  one  will 
be  a  SISO  “reference  document”  in  support  of  the  two  other  ones. 

The  final  technical  report  published  under  an  RTO  reference  will  be  a  short  introduction  to  the  SISO 
documents. 

Overall  Duration 

This  TG  should  be  established  for  the  full  life-cycle  of  the  SISO  GM  V&V  PDG  and  no  longer  than  3  years. 


RESOURCES 

Membership 

Participating  Nations  are  initially  France  (lead  Nation),  Canada,  Denmark,  Netherlands  and  Sweden. 
All  NMSG  Nations  are  invited  to  join. 

Anticipated  Meeting  Schedule 

The  TG  is  expected  to  meet  face-to-face  2-3  times  a  year  (1-3  day  meetings).  It  will  use  teleconferencing 
as  far  as  possible.  Face-to-face  meetings  could  be  organized  during  or  in  conjunction  of  SISO  workshops  and 
other  international  events  either  in  North  America  or  in  Europe. 
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National  and/or  NATO  Resources  Needed 

Input  to  and  participation  in  the  meetings  including  travel  funds  (1-2  specialists  per  Nation). 


SECURITY  CLASSIFICATION  LEVELS 

Unclassified/  Unlimited. 


PARTICIPATION  BY  PARTNER  NATIONS 

Partner  Nations  are  invited  to  participate. 


LIAISON 

ACT,  NIAG,  NCSA,  SISO,  US  M&S  CO. 
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From:  Martin  ADELANTADO  (NATO  MSG-073  Chairman)  ONERA-MiP/DTIM 
BP74025 

2  avenue  Edouard  Belin 
FR-31055  TOULOUSE  CEDEX  4 
France 

Phone:  +33  (0)5  62  25  26  61 
Email:  Martin.Adelantado@onera.fr 

To:  Mr.  David  DRAKE, 

SISO  Board  of  Directors  Chair 


Dear  David, 

As  you  know  the  NATO  NMSG-073  task  Group  on  «  GM-VV:  Generic  Methodology  for  Verification  and 
Validation  »  is  supporting  the  process  of  SISO  standardization  of  this  methodology  since  3  years. 
Recently,  the  PDG  meeting  held  in  Orlando  on  Thursday  29  March  2012  has  approved  the  balloting 
resolutions  proposed  by  the  GM-VV  PDG  Drafting  Group  chaired  by  Manfred  Roza  (NLR  -  NED). 
The  balloted  document  is  entitled  «  GM-VV  Vol.  1:  Introduction  and  Overview  »  referenced  by 
SISO-GUIDE-00 1.1-2012. 

As  Chairman  of  the  NATO  NMSG-073  and  in  agreement  with  Jean-Louis  IGARZA,  GM-VV  PDG 
Chairman,  I  am  writing  you  to  request  permission  to  include  parts  of  the  document 
«SISO-GUIDE-00 1.1-2012  »  in  the  NMSG-073  Task  Group  Final  Report.  I  believe  that  SISO  is  currently 
the  holder  of  the  copyright.  If  you  do  not  currently  hold  the  rights,  please  provide  me  with  any  information 
that  can  help  me  contact  the  proper  rights  holder.  Otherwise,  your  permission  confirms  that  you  hold  the 
right  to  grant  this  permission. 

This  request  is  for  a  non-exclusive,  irrevocable,  and  royalty-free  permission,  and  it  is  not  intended  to 
interfere  with  other  uses  of  the  same  work  by  you.  I  would  be  pleased  to  include  a  full  citation  to  your 
work  and  other  acknowledgement  as  you  might  request. 

1  would  greatly  appreciate  your  permission.  If  you  require  any  additional  information,  do  not  hesitate  to 
contact  me  by  mail  or  at  the  phone  number  above. 

Would  you  therefore: 

a)  Grant  permission  for  the  use  as  described  above:  granted/not  granted 

If  you  agree  with  the  terms  as  described  above,  please  indicate  your  permission  by  signing  below  and 
return  the  signed  request  to  me  in  the  enclosed  self-addressed  stamped  envelope.  1  have  included  a  second 
copy  for  your  records. 


Name:  David  L.  Drake 
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Title:  SISO  Board  of  Directors  Chair 
Date:  19  April  2012 

b)  List  any  conditions  attaching  to  publication  in  addition  to  those  described  above, 

c)  Provide  the  address  of  the  copyright  owner  if  you  do  not  hold  these  rights: 

1.  Not  applicable 

d)  Give  the  full  text  and  preferred  location  of  any  acknowledgement  required: 

Copyright  2012  by  Simulation  Interoperability  Standards  Organization  -  SISO 
Sincerely, 


Martin  Adelantado 


19  April  2012 
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